Image sensor under the display

Abstract

A device includes a display and a first light source configured to emit light, wherein the first light source is proximate to the display. The device also includes a first camera disposed behind the display, wherein the first camera is configured to detect a reflection of the light emitted by the first light source. The first camera is also configured to capture a first image based at least in part on the reflection, wherein the reflection is partially blocked by the display. The device also includes a second camera proximate to the display, wherein the second camera is configured to capture a second image. In addition, the device includes a depth map generator configured to generate depth information about one or more objects in a field of view (FOV) of the first camera and the second camera based at least in part on the first image and the second image.

Description

Technical Field

[0001] The present embodiments relate generally to systems and methods for imaging, and particularly to image sensors beneath displays. Background Art

[0002] Many electronic devices, such as smartphones, tablets, laptops, and other computing systems, include a camera and a display. The camera is typically positioned in an area of ​​the device that provides an unobstructed view of the environment. Therefore, when integrated on the same surface as the display, the camera is typically positioned adjacent to the display, within a notch or cutout. This often results in a large, unsightly black border around the display, which can detract from the appearance of the device and limit the screen-to-body ratio of the device. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] A device is disclosed. The device may include a display and a first light source configured to emit light, wherein the first light source is close to the display. The device may also include a first camera disposed behind the display, wherein the first camera is configured to detect a reflection of light emitted by the first light source. The first camera may also be configured to capture a first image based at least in part on the reflection, wherein the reflection is partially blocked by the display. The device may also include a second camera close to the display, wherein the second camera is configured to capture a second image. In addition, the device may include a depth map generator configured to generate depth information about one or more objects in a field of view (FOV) of the first camera and the second camera based at least in part on the first image and the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0006] Figure 1 A block diagram of an electronic device in accordance with some embodiments is shown.

[0007] Figure 2 A block diagram of an electronic device in accordance with some embodiments is shown.

[0008] Figure 3A An example electronic system is shown in accordance with some embodiments.

[0009] Figure 3B An example electronic system is shown in accordance with some embodiments.

[0010] Figure 3C An example electronic system is shown in accordance with some embodiments.

[0011] Figure 3D An example electronic system is shown in accordance with some embodiments.

[0012] Figure 4 A cross-section of an example electronic system is shown in accordance with some embodiments.

[0013] Figure 5 A display of an electronic system is shown in accordance with some embodiments.

[0014] Figure 6 An example image captured by a camera disposed below a display is shown in accordance with some embodiments.

[0015] Figure 7 Example arrangements of display pixels and / or display sub-pixels and optical sensing elements are shown according to some embodiments.

[0016] Figure 8 A block diagram of an image processing system according to some embodiments is shown.

[0017] Fig. 9 An illustrative flow chart depicting example operations for generating depth information about an object or scene in accordance with some embodiments is shown.

[0018] Fig. 10A An example image captured by a camera below the display is shown.

[0019] Fig. 10B Shows the filtered Fig. 10A .

[0020] Fig.11 An example 3D image is shown in accordance with some embodiments.

[0021] Fig.12 An example application of depth information in accordance with some embodiments is shown. DETAILED DESCRIPTION

[0022] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" refers to being directly connected to or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required for practicing the example embodiments. In other cases, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure unclear. Some parts of the following detailed description are presented according to the process, logic blocks, processing and other symbolic representations of the operation of the data bits in the computer memory. The interconnection between circuit elements or software blocks can be displayed as a bus or a single signal line. Each bus can alternatively be a single signal line, and each single signal line can alternatively be a bus, and a single line or bus can represent any one or more of the countless physical or logical mechanisms for communication between components.

[0023] Unless explicitly stated otherwise (as will be apparent from the following discussion), it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0024] Unless explicitly described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include packaging materials.

[0025] The non-transitory processor-readable storage medium may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0026] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors. The term "processor" as used herein may refer to any general purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0027] Aspects of the present disclosure provide a system and method for imaging that can be implemented behind a display of an electronic system or device. In some embodiments, the electronic system may include a display, at least one illuminator (also referred to herein as a "first light source"), and an imaging system including two or more cameras (also referred to herein as "camera sensors"). In some embodiments, the display may include a multi-hole display, such as, for example, an organic light emitting diode (OLED) display or a micro light emitting diode (micro-LED) display. The illuminator may include a NIR light emitting diode (LED) or other light source capable of emitting wavelengths of light in the infrared (IR) spectrum. The camera may be configured to detect light in the IR spectrum (and these cameras are also referred to herein as "NIR cameras"). In addition, the cameras may be separated by a threshold distance. The separation distance between the cameras enables the imaging system to collect depth information about (one or more) objects in the FOV of the camera. For example, each camera can capture an image of an object, and depth information about the object can be extracted from the image using stereopsis.

[0028] In some embodiments, the camera may be disposed below the display. In this configuration, the FOV of the camera may be partially obscured by display pixels and / or display sub-pixels in the display. However, aspects of the present disclosure recognize that the display may include "holes" or blank spaces between display pixels and / or display sub-pixels for at least some light to pass through. In some embodiments, a neural network model (e.g., a convolutional neural network (CNN)) may be used to filter out noise or interference, such as the "screen door effect" that may be caused by display pixels and / or display sub-pixels in the display. The resulting image is therefore suitable for depth sensing and / or other image processing. Among other advantages, this eliminates the need for unsightly black borders, cutouts, or notches in the display bezel. In addition, the screen-to-body ratio of the electronic system can be improved, and the electronic system can support an infinite display.

[0029] Figure 1 A block diagram of an electronic device 100 (also referred to as “device 100 ”) is shown in accordance with some embodiments. The electronic device 100 includes an illuminator 108 , a display 102 , a first camera 104 , a second camera 106 , and a depth map generator 134 .

[0030] The illuminator 108 is configured to illuminate an object or scene being imaged. In some embodiments, the illuminator 108 can be configured to emit IR light (e.g., NIR light). In addition, the illuminator 108 can include an LED (e.g., a NIR LED), or the illuminator 108 can include a display pixel and / or a display sub-pixel in the display 102. In addition, in some embodiments, the device 100 can include more than one illuminator 108.

[0031] The display 102 is configured to display visual information to a user, receive user input, and / or transmit light from an object or scene being imaged to the first camera 104 and the second camera 106. The display 102 may include a porous display, such as an OLED display or a micro-LED display, which includes holes or gaps between display pixels and / or display sub-pixels. In some embodiments, the display 102 may include a transparent layer configured to transmit light from an object or scene being imaged to the first camera 104 and the second camera 106.

[0032] Each of the first camera 104 and the second camera 106 is configured to capture an image of an object or scene in front of the display 102. The first camera 104 and the second camera 106 may each include an active pixel sensor array (e.g., a photodiode, a complementary metal oxide semiconductor (CMOS) image sensor array, a charge coupled device (CCD) array), and / or any other sensor capable of detecting wavelengths of light in the visible spectrum, the IR spectrum, or the ultraviolet spectrum. In some embodiments, each of the first camera 104 and the second camera 106 may be configured to detect light in the IR or non-visible spectrum. In addition, each of the first camera 104 and the second camera 106 may be configured to filter or reject visible light emitted and / or transmitted by the display 102. In other embodiments, both the first camera 104 and the second camera 106 may be configured to detect visible light, or the first camera 104 may be configured to detect NIR light and the second camera 106 may be configured to detect visible light.

[0033] like Figure 1 As shown in , each of the first camera 104 and the second camera 106 is disposed behind (or proximate to) the display 102. With this configuration, light from an object or scene being imaged may be refracted or bent as it passes through the display 102 to reach the cameras 104 and 106. Additionally, display pixels and / or display sub-pixels in the display 102 may partially obstruct the camera's FOV. However, each of the first camera 104 and the second camera 106 may still image an object or scene through the display 102 because, as discussed above, the display 102 is porous, which permits some light to pass through the display 102 to reach the camera. Additionally, each of the first camera 104 and the second camera 106 may be configured to perform wide-range imaging, where an object or scene is imaged at close range or at a distance. Note that although in Figure 1 Two cameras are depicted in the example of , but other embodiments of electronic device 100 may include only one camera, or embodiments may include more than two cameras.

[0034] The depth map generator 134 may be configured to determine depth information about an object or scene imaged by the first camera 104 and the second camera 106. More specifically, the depth map generator 134 may be configured to receive images captured by the first camera 104 and the second camera 106, and the depth map generator 134 may combine the images using stereopsis to extract depth information from the images.

[0035] During operation, the illuminator 108 may emit light 116 to illuminate a scene in front of the first camera 104 and the second camera 106. Objects in the scene may reflect at least a portion of the light 116 back toward the display 102, for example, as reflected light 116R(1) and 116(R2). Some of the reflected light 116R(1) and 116R(2) may be blocked by display pixels and / or display sub-pixels in the display 102, while the remaining portion of the reflected light 116R(1) and 116R(2) passes through holes or gaps between the display pixels and / or display sub-pixels. The reflected light 116R(1) and 116R(2) that passes through the holes may be detected by the first camera 104 and the second camera 106, respectively. The first camera 104 may capture the reflected light 116R(1) detected as the first image 144, and the second camera 106 may capture the reflected light 116R(2) detected as the second image 166. Furthermore, the depth map generator 134 may receive the first image 144 and the second image 166 and combine the images to extract depth information 135 about the object.

[0036] Figure 2 1 shows a block diagram of an electronic device 200 (also referred to as “device 200”) according to some embodiments. The electronic device 200 may be Figure 1 An embodiment of the electronic device 100. Figure 2 , the first camera 204 is disposed behind the display 202 and the second camera 206 is disposed proximate to the display 202 (or disposed in the same plane as the display 202 and the illuminator 208). In some embodiments, the first camera 204 may be a NIR camera and the second camera 206 may be configured to detect light in the visible spectrum (e.g., including red, green, and blue (RGB) color components of light) ("RGB camera"). In other embodiments, the second camera 206 may be a NIR camera.

[0037] During operation, illuminator 208 may emit light 216 to illuminate a scene in front of first camera 204 and second camera 206. Objects in the scene may reflect at least a portion of light 216 back toward display 202, e.g., as reflected light 216(R)(1) and 216(R)(2). Figure 2 As shown in FIG. 2 , reflected light 216 (R) (1) (and / or light from the sun or another illuminator) is detected by the second camera 206 and captured in the second image 266. In addition, reflected light 216 (R) (2) may be blocked by display pixels and / or display sub-pixels in the display 202, while the remaining portion of reflected light 216 (R) (2) passes through holes or gaps between display pixels and / or display sub-pixels. The reflected light 216 (R) (2) detected by the first camera 204 is captured in the first image 244. As described above with reference to FIG. Figure 1As described above, depth map generator 234 may receive and combine first image 244 and second image 266 to extract depth information 235 about an object.

[0038] Figure 3A An example electronic system 300A is shown in accordance with some embodiments. The electronic system 300A may be Figure 1 and Figure 2 Embodiments of the electronic device 100 and / or 200. Figure 3A As shown in FIG. 3 , electronic system 300A includes display 302 , and cameras 304 and 306 .

[0039] Display 302 may include a porous display, such as an OLED display or a micro-LED display, that includes holes or gaps between display pixels and / or display sub-pixels. Each of cameras 304 and 306 may include an active pixel sensor array (e.g., a photodiode, a CMOS image sensor array, a CCD array, and / or any other sensor capable of detecting wavelengths of light in the visible, infrared, or ultraviolet spectrum). In addition, cameras 304 and 306 may be configured to perform wide-range imaging, where objects or scenes are imaged at close range or at long range. Note that while in Figure 3A Although two cameras are depicted in the example of FIG. 3 , other embodiments of electronic system 300A may include more than one camera. Figure 3A In some embodiments, each of cameras 304 and 306 can be configured to support depth sensing by capturing images of an object. The images can then be combined using stereopsis to perceive the depth and / or three-dimensional (3D) structure of the object. It is also noted that during operation, electronic system 300A can be positioned or mounted in a vertical, horizontal, or diagonal orientation relative to the scene or object being imaged.

[0040] In some embodiments, cameras 304 and 306 may be disposed below display 302. With this configuration, display pixels and / or display sub-pixels in display 302 may partially obstruct the FOV of the camera. However, because display 302 is porous, cameras 304 and 306 may still image an object or scene through display 302. In other words, display 302 may include holes or empty spaces between pixels and / or sub-pixels. When light is emitted from a light source (such as illuminator 308 described further below), the light may be reflected from an object or feature in front of electronic system 300A and then pass through the holes or empty spaces in display 302 to cameras 304 and 306. In some embodiments, at least one of cameras 304 and / or 306 is a NIR camera configured to detect light in the IR or non-visible spectrum. The NIR camera may filter or reject visible light, such as any light emitted by display 302. In some other embodiments, one of cameras 304 or 306 may be an RGB camera.

[0041] It is noted that because pixels in display 302 will at least partially block light from passing to cameras 304 and / or 306, images captured by the cameras will contain noise or interference (e.g., a "screen door effect"). In some embodiments, a neural network model (e.g., a CNN) may be used to filter out noise or distortion in images captured by cameras 304 and 306. For example, a CNN may be trained to identify or generate inferences about a screen door effect caused by display pixels and / or display sub-pixels of display 302. By removing the screen door effect, the resulting image may be suitable for depth sensing and / or further image analysis.

[0042] Figure 3B An example electronic system 300B is shown according to some embodiments. The electronic system 300B may be a reference Figure 1 , 2 3A and the electronic device 100 and / or 200, and / or the electronic system 300A described in the embodiments. Figure 3B , illuminator 308 can be positioned near a corner of electronic system 300B. In addition, illuminator 308 can be disposed below display 302, in the same plane as display 302, in a bezel of display 302, or in any combination thereof. In some embodiments, illuminator 308 can be configured to illuminate an object or scene being imaged. For example, illuminator 308 can include an LED (e.g., a NIR LED), display pixel, and / or display sub-pixel configured to illuminate an object or scene using IR light.

[0043] Figure 3C An example electronic system 300C is shown according to some embodiments. The electronic system 300C may be a reference Figure 1 ,2 , 3A and 3B describe embodiments of the electronic device 100 and / or 200, and / or the electronic system 300A and / or 300B. Figure 3C As shown in FIG. 3 , cameras 304 and 306 may be disposed in a middle region of electronic system 300C along the long axis. Figure 3B In contrast, Figure 3C The cameras 304 and 306 are separated by a smaller distance. Figure 3C The maximum distance that cameras 304 and 306 can sense depth may be greater than Figure 3B The maximum distance at which the camera can sense depth is limited. Note also that Figure 3C , flanking each of the cameras 304 and 306 are two illuminators 308. The illuminators 308 may be disposed below the display 302, in the same plane as the display 302, in a bezel of the display 302, or any combination thereof. In some embodiments, the illuminators 308 may be configured to illuminate an object or scene being imaged. For example, the illuminators 308 may include NIR LEDs configured to illuminate an object or scene using IR light.

[0044] Figure 3D An example electronic system 300D is shown according to some embodiments. The electronic system 300D may be a reference Figure 1 , 2 , 3A-3C described in the electronic device 100 and / or 200, and / or the electronic system 300A-300C. Figure 3D As shown in FIG. 3 , cameras 304 and 306 may be disposed in a middle region of electronic system 300D along the short axis. Figure 3C In contrast, Figure 3D The cameras 304 and 306 are separated by a shorter distance. Figure 3D The maximum distance that cameras 304 and 306 can sense depth may be greater than Figure 3C The maximum distance at which the camera can sense depth is limited. Note also that Figure 3D 300D and toward the center of the electronic system 300D. The illuminator 308 can be disposed below the display 302, in the same plane as the display 302, in a bezel of the display 302, or in any combination thereof. In some embodiments, the illuminator 308 can be configured to illuminate an object or scene being imaged. For example, the illuminator 308 can include a NIR LED configured to illuminate an object or scene using IR light.

[0045] Figure 4 4 shows a cross-section of an electronic system 400 according to some embodiments. Figure 4As shown in FIG. 4 , the electronic system 400 includes a display layer 402, an illuminator 408, and a camera module 410. The electronic system 400 may be related to Figure 1 and 3A -3D depicts example embodiments of electronic devices 100 and / or electronic systems 300A-300D.

[0046] The display layer 402 may include a multi-layer porous display, such as an OLED display or a micro-LED display. Figure 4 As shown in , display layer 402 may include display surface 414 and emissive layer 418. In some embodiments, emissive layer 418 may include a plurality of display pixels and / or display sub-pixels with holes, gaps, or empty spaces between each display pixel and / or display sub-pixel. In some embodiments, at least one of display layers 402 may include a transparent layer configured to transmit light from an object or scene being imaged to cameras 404 and 406. In addition, as shown in FIG. Figure 4 As shown in , light from an object or scene being imaged travels along optical path 416 and may be refracted or bent as it passes through display layers 402 and lens 412 to cameras 404 and 406 .

[0047] In some embodiments, illuminator 408 can be positioned adjacent to display layer 402. Illuminator 408 can be disposed below display layer 402, in the same plane as display layer 402, in a bezel of electronic system 400, or any combination thereof. Illuminator 408 can be configured to illuminate an object or scene being imaged. For example, illuminator 408 can include an LED (e.g., a NIR LED), display pixels, and / or display sub-pixels configured to illuminate an object or scene using IR light.

[0048] like Figure 4 As shown in , camera module 410 is disposed below display layer 402 and may include cameras 404 and 406, and lens 412. Cameras 404 and 406 may include NIR CMOS sensors configured for wide-range imaging, where objects or scenes are imaged at close range or at long distances. Cameras 404 and 406 may also be configured to quickly sense objects such as fingers, hands, and heads. In addition, because electronic system 400 includes two cameras, cameras 404 and 406 may be configured to support depth sensing. The resolution of depth sensing may depend on the separation distance D between cameras 404 and 406. In other words, the greater the distance D, the greater the amount of depth information that can be derived from cameras 404 and 406. In some embodiments, cameras 404 and 406 may be configured to operate in multiple modes (such as low power mode to support ambient light sensing (ALS) function).

[0049] During operation, illuminator 408 can illuminate an object or scene in front of electronic system 400. For example, illuminator 408 can emit IR light, which reflects from the object or scene and then travels back toward electronic system 400. The travel path of the reflected IR light is Figure 4 Depicted as optical path 416. Figure 4 , optical path 416 passes through display surface 414 to emissive layer 418. In emissive layer 418, some of the reflected IR light may be blocked by display pixels and / or display sub-pixels of display layer 402, but at least some of the reflected IR light may pass through holes or empty spaces between display pixels and / or display sub-pixels. The remaining IR light then travels through the remaining display layer 402 and lens 412 to cameras 404 and 406. Each of cameras 404 and 406 can capture the reflected IR light as a separate image.

[0050] Aspects of the present disclosure recognize that by positioning cameras 404 and 406 below display layer 402, images captured by cameras 404 and 406 may include a "screen door effect" ( Figure 6 ), distortion, or other artifacts caused by IR light passing through the display layer 402. As described in more detail below, a neural network model (e.g., CNN) can be used to filter such distortion. Once the captured image is filtered, depth information can be extracted from the captured image using stereoscopy. In some aspects, the depth information can be used to construct a depth map. In some other embodiments, the depth information can be used to generate a 3D image. Further, in some aspects, the depth information can be used for biometric identification, authentication, or tracking, or for other applications.

[0051] Figure 5 5 shows a display of an electronic system 500 according to some embodiments. The electronic system 500 may be as described with respect to Figure 1 , 2 , 3A-3D and 4 describe example embodiments of the electronic device 100 and / or 200, and / or the electronic system 300A-300D or 400.

[0052] Electronic system 500 may include display 502. Display 502 may be a porous display, including but not limited to an OLED display or a micro-LED display. In other words, display 502 may include display pixels and / or display sub-pixels (in the form of a plurality of pixels) separated by holes, gaps, or empty spaces. Figure 5 as a gray rectangle). Figure 5As shown in , the pattern of display pixels and / or display sub-pixels can vary over a limited area 522A of the display 502. However, aspects of the present disclosure recognize that the pattern can repeat over larger areas 522B and 522C of the display 502. Aspects of the present disclosure further recognize that due to the variation in the pattern of display pixels and / or display sub-pixels, the distortion in the captured image may appear random and / or inconsistent. However, by placing an optical sensing element (e.g., Figure 7 If the optical sensing elements 704 of the display are aligned with the repeating pixel pattern, the variation in distortion will be the same or substantially similar across each optical sensing element. It is noted that the display pixels and / or display sub-pixels may vary in shape and size, and they may be arranged in rows and columns, in a circular configuration, or in another configuration.

[0053] Figure 6 An example image 600 is shown captured by a camera disposed below a display according to some embodiments. Image 600 may be captured by a camera disposed behind or below a porous display, such as, for example Figure 1 , 2 , 3A-3D and 4 of the camera 104, 106, 204, 304, 306, 404 or 406. Figure 6 As shown in , image 600 includes a grid of dark lines and bright spots - or a "screen door effect". The dark lines correspond to a pattern of display pixels and / or display sub-pixels, similar to Figure 5 . Dark lines are produced by display pixels and / or display sub-pixels that block some IR light from passing through the display. In contrast, brighter spots correspond to gaps or holes between display pixels and / or display sub-pixels. Bright spots are produced by IR light that passes through the holes or gaps in the display to the camera below.

[0054] In some embodiments, the image captured by the camera below the display may include a grid or screen door effect, which appears as if the camera below the display is simultaneously imaging through multiple layers of screen doors (or stacks of screen doors), where each screen door layer has a different pitch (i.e., hole size or density). In such embodiments, IR light may pass through the holes (or gaps) between display pixels and / or display sub-pixels of the screen door layers. Furthermore, each point of IR light is diffused into different locations horizontally and vertically (i.e., as a point spread function), and each point of IR light is diffused across the image in a similar manner, which may result in significant distortion.

[0055] like Figure 6As shown in , portions of image 600 are obscured by display pixels and / or display sub-pixels of the display. In other words, the display pixels and / or display sub-pixels may reduce the amount of IR light available for sensing and effectively block the FOV of the camera below the display. Therefore, it may be desirable to filter or calibrate the image captured by the camera below the display in order to eliminate screen door effects, distortion, or other artifacts.

[0056] Figure 7 An example arrangement 700 of display pixels and / or display sub-pixels 702 ("display pixels 702") and optical sensing elements 704 is shown in accordance with some embodiments. Each of the display pixels 702 may be Figure 4 , 5 and / or the exemplary embodiment of the display pixel shown in 6. The optical sensing elements 704 may be, respectively or collectively, such as, for example, Figure 1 , 2 , 3A-3D and 4 describe embodiments of cameras 104, 106, 204, 304, 306, 404 or 406.

[0057] In some embodiments, display pixel 702 may be part of a multi-aperture display such as, for example, an OLED display or a micro-LED display. Each optical sensing element 704 may include a NIR CMOS sensor, and each optical sensing element 704 may have a corresponding micro-lens 712. Figure 7 Only six display pixels 702 are shown in the figure, but in actual embodiments, the imaging system 700 under the display may include nearly any number of display pixels 702 (e.g., hundreds, thousands, millions, or more), and a corresponding number of optical sensing elements 704 and micro lenses 712.

[0058] exist Figure 7 In the example of FIG. 7 , display pixels 702 can be aligned with optical sensing elements 704. In some embodiments, each optical sensing element 704 can be aligned with a display area (such as a Figure 5 Due to the alignment, a screen door effect or other distortion may appear as a consistent or repeating pattern in the image captured by the optical sensing element 704. Aspects of the present disclosure recognize that repeating patterns of distortion may be more easily filtered out from the captured image. For example, a neural network model may be more easily trained to recognize repeating patterns in an image and may provide more robust filtering of the screen door effect when the pattern repeats.

[0059] Aspects of the present disclosure further recognize that when an object is imaged in a bright or sunny environment, the display pixels 702 may become saturated or washed out. Therefore, some display pixels may not be suitable for imaging in both bright and dark environments. In some embodiments, the display pixels 702 may include non-linear or logarithmic pixels. Because the charge stored by such logarithmic pixels varies nonlinearly with respect to the brightness or intensity of light, the display pixels 702 may be well suited for imaging in both bright and dark conditions.

[0060] Figure 8 8 shows a block diagram of an image processing system 800 according to some embodiments. The image processing system 800 includes a device interface 810, a processor 820, and a memory 830. For the purposes of discussion herein, the processor 820 is Figure 8 800 is shown as being coupled to a device interface 810 and a memory 830. For practical embodiments, the device interface 810, the processor 820, and / or the memory 830 may be connected together using one or more buses (not shown for simplicity). Note that in some embodiments, the image processing system 800 may be the above-mentioned Figure 1 , 2 , 3A-3D, 4 and 5 describe any of the electronic devices 100 and / or 200, and / or the electronic systems 300A-300D, 400, and / or 500, and an application specific integrated circuit (ASIC) or other integrated circuit (IC).

[0061] The device interface 810 can transmit signals to and receive signals from input devices, including but not limited to one or more cameras (not shown for simplicity) and / or displays. In some embodiments, the device interface 810 may include camera interfaces 814 and 816. Each camera interface 814 and 816 can be used to communicate with different cameras. For example, the first camera interface 814 can transmit an activation signal to the first camera and receive sensor information from the first camera to capture an image of an object or scene. In addition, the second camera interface 816 can transmit an activation signal to the second camera and receive sensor information from the second camera to capture an image of the same object and / or scene. In some embodiments, the device interface 810 may also include a display interface 812. The display interface 812 can be used to communicate with a display and / or provide a visual interface to a user of the image processing system 800.

[0062] The memory 830 may include an image buffer 831 to store images received via the camera interface 814 and / or 816. The memory 830 may also include a non-transitory computer-readable medium (e.g., one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, hard drive, etc.), which may store at least the following SW modules:

[0063] • An image filtering SW module 832 for filtering images received via the camera interface 814 and / or 816, the image filtering SW module 832 further comprising:

[0064] ◦ A neural network model 833 for filtering, reducing, or removing noise from images received via camera interfaces 814 and / or 816; and

[0065] A depth map SW module 834 for determining depth information of objects or features in images received via the camera interface 814 and / or 816, the depth map SW module 834 further comprising:

[0066] ◦ A stereoscopic submodule 835 for extracting depth information from a pair of images received via respective camera interfaces 814 and 816;

[0067] o a 3D imaging submodule 836 for generating a 3D image based at least in part on the depth information; and

[0068] ◦ Biometric Identification, Authentication, and Tracking (BIAT) submodule 837 for identifying, authenticating, and / or tracking a subject or object in an image based at least in part on depth information.

[0069] Each SW module includes instructions that, when executed by the processor 820 , cause the image processing system 800 to perform a corresponding function.

[0070] For example, the processor 820 may execute an image filtering SW module 832 (or "image filter") to filter images received via the camera interfaces 814 and / or 816. When executing the image filtering SW module 832, the processor 820 may use a neural network model 833 to filter, reduce, or eliminate noise (such as a screen door effect) from the images received via the camera interfaces 814 and / or 816. The processor 820 may further execute a depth map SW module 834 to determine depth information of objects or features in the images received via the camera interfaces 814 and / or 816. When executing the depth map SW module 834, the processor 820 may use a stereoscopic submodule 835, a 3D imaging submodule 836, and / or a BIAT submodule 837. For example, the processor 820 may execute the stereoscopic submodule 835 to extract depth information from a pair of images received via the respective camera interfaces 814 and 816. The processor 820 may also execute the 3D imaging submodule 836 to generate a 3D image based at least in part on the depth information. Processor 820 may also execute BIAT submodule 837 to identify, authenticate, and / or track a subject or object in an image based at least in part on depth information.

[0071] Fig. 9 An illustrative flow chart depicting example operations 900 for generating depth information about an object or scene in accordance with some embodiments is shown. Figure 1 , operation 900 may be performed by the electronic device 100 .

[0072] The electronic device 100 may receive a first image of an object captured by a first camera disposed behind an electronic display (910). The first image may be based at least in part on reflections of light (eg, NIR light) emitted by an illuminator (or first light source).

[0073] The electronic device 100 may also receive a second image of the object captured by a second camera proximate to the electronic display (920). In some embodiments, the second camera may be disposed in the same plane as the electronic display and the illuminator, such as Figure 2 as shown in .

[0074] After the electronic device 100 receives the first image and the second image, the electronic device 100 may process the images to generate depth information (930). In some embodiments, one or more neural network models may be used to filter noise or distortion in the first image and the second image. As described above, the depth information may indicate the distance or depth of the object in the FOV of the first camera and the second camera, and the depth information may be based at least in part on the first image and the second image. In some embodiments, the electronic device 100 may combine the first image and the second image and extract depth information from the image about the object.

[0075] Fig. 10A An example image 1010 captured by a camera below a display is shown. Figure 8 , image 1010 can be transmitted from a camera below the display to, for example, camera interface 814. Camera interface 814 can then transmit image 1010 to image buffer 831 for storage. Fig. 10A As shown in , image 1010 is blurred by the screen door effect. Therefore, it may be desirable to filter image 1010 to remove the distortion.

[0076] Fig. 10B An example image 1020 is shown, which is a filtered Fig. 10A To generate image 1020, refer to Figure 8 , the image buffer 831 may transmit the image 1010 to the image filtering SW module 832. The image filtering SW module 832 may use the neural network model 833 to filter out the screen door effect from the image 1010. The resulting image is the image 1020, as shown in FIG. Fig. 10B The middle diagram shows the person more clearly.

[0077] In some embodiments, the neural network model 833 may include a CNN trained to filter out the screen door effect or other noise from the captured image. In other words, the CNN can be trained to infer the original or clean image of the object or scene from the blurred image of the object or scene. In some embodiments, the CNN can be trained by receiving and processing at least one original image of the object or scene, and at least one blurred image of the object or scene. In some embodiments, instead of or in addition to the CNN, an inverse point spread function transform can be used to filter the image. Therefore, the image filtering SW module 832 can use the neural network model 833 and / or the inverse point spread function transform to filter the image from the camera below the display.

[0078] Fig.11 An example 3D image 1110 is shown according to some embodiments. To generate the example 3D image 1110, refer to Figure 8 , each of the cameras under the two displays can capture an image of a group of people at the same time, and can transmit the captured images to camera interfaces 814 and 816, respectively. Camera interfaces 814 and 816 can then transmit the images to image buffer 831 for storage. Subsequently, image buffer 831 can transmit the images to image filtering SW module 832, which can remove the screen door effect or any other distortion from the images using neural network model 833. Depth map SW module 834 can then use stereoscopic observation submodule 835 to combine the filtered images and extract depth information from the images.

[0079] In some embodiments, instead of stereoscopy, or in addition to stereoscopy, the depth map SW module 834 may use other methods to determine depth information from the pair of images. Figure 4 and Figure 8 , the camera module 410 may use special optics, and the depth map SW module 834 may use special algorithms with diffractive optics, multiple apertures or multiple FOVs (where the aperture or FOV of one camera is different from the aperture or FOV of another), or coded or phase apertures to extract depth information. As another example, the depth map SW module 834 may use one camera to sense one wavelength range (e.g., visible light) and another camera to sense another wavelength range (e.g., IR light). The depth map SW module 834 may then triangulate the images captured by the two cameras to extract depth information.

[0080] Once the depth information has been extracted, the 3D imaging submodule 836 may generate a Fig.113D image 1110. In some embodiments, one of the two images processed by the 3D imaging submodule 836 may include color (or RGB) information from a standard optical RGB camera. The 3D imaging submodule 836 may overlay the color information onto the 3D image to generate a color 3D image.

[0081] In some embodiments, the 3D imaging submodule 836 may be configured to project a 3D image onto a display via the display interface 812. The 3D imaging submodule 836 may also be configured to modify or rotate the 3D image, thereby allowing the user to interact with the 3D image. For example, in some aspects, the 3D imaging submodule 836 may be configured to provide a bokeh focus function. In other words, when a 3D image is projected onto a display via the display interface 812, the user may be able to determine which (one or more) parts of the 3D image appear in focus, and which (one or more) parts of the 3D image appear blurred or out of focus. For any (one or more) blurred parts, the user may be able to adjust the degree or type of blur.

[0082] Fig.12 FIG. 4 shows an example application of depth information according to some embodiments. Figure 8 , the depth map SW module 834 may use the BIAT submodule 837 to identify, authenticate, and / or track a subject or object in an image based at least in part on the depth information. Fig.12 As shown in , each of cameras A and B can capture an image of a person during imaging 1210. As described above, image filtering SW module 832 can remove distortion from the image using neural network model 833, and depth map SW module 834 can extract depth information from the image using stereoscopic submodule 835. 3D imaging submodule 836 can then generate a 3D image, such as 3D image 1220, based at least in part on the depth information from the image.

[0083] In some embodiments, the BIAT submodule 837 can identify any subject, object, or feature in the 3D image, such as, for example, a person in the 3D image 1220. For example, the BIAT submodule 837 can compare the 3D image 1220 to stored data (e.g., reference images of people and / or users). In the event that the BIAT submodule 837 determines a match between the 3D image 1220 and the stored data, the BIAT submodule 837 can authenticate or verify the identity of the person in the 3D image 1220 or simply determine that the person is present in the 3D image 1220. Thus, the BIAT submodule 837 can be used to perform biometric identification and / or authentication based at least in part on depth information.

[0084] In some embodiments, the BIAT submodule 837 can also be configured to perform biometric tracking. Fig.12 , the 3D image 1220 may correspond to two images captured by the camera at a first time instance. The BIAT submodule 837 may determine the position of the person in the 3D image 1220 and then store the position in, for example, the image buffer 831. The BIAT submodule 837 may then receive a 3D image based on the two images captured by the camera at a second time instance, where the second time instance is subsequent to the first time instance. The BIAT submodule 837 may then identify and / or authenticate the person in the 3D image corresponding to the second time instance. If the BIAT submodule 837 determines that the person in the 3D image corresponding to the second time instance is the same as the person in the 3D image 1220 corresponding to the first time instance, the BIAT submodule 837 may determine the position of the person in the 3D image corresponding to the second time instance and store the position in, for example, the image buffer 831. The BIAT submodule 837 may then use the stored positions of the 3D images corresponding to the first and second time instances to track the position or movement of the person over time. In other embodiments, the BIAT submodule 837 can be configured to track objects such as eyes, fingers, or hands, and / or the BIAT submodule 837 can be configured to perform gesture recognition and / or air touch. The BIAT submodule 837 can also be configured to recognize and / or track biometric indicators such as heartbeat (i.e., pulse) or blood pressure.

[0085] In some aspects, the BIAT submodule 837 can perform more robust identification, authentication, and / or tracking by rotating and / or changing the distance of a subject or object in a 3D image before comparing the 3D image to stored data. It should be noted that some authentication algorithms require the user's face to be positioned at a specific orientation and / or position of the camera's FOV (e.g., directly facing the camera) for proper analysis. Therefore, reference Fig.12 1220, the BIAT submodule 837 can be configured to rotate the 3D image 1220 so that the face of the person in the 3D image 1220 conforms to a desired orientation (e.g., an orientation similar to the orientation of the person in the reference image), such as the orientation of the 3D image 1230. The BIAT submodule 837 can then analyze specific data points in the 3D image 1230 (such as the general alignment of the person's eyes, nose, mouth, and / or other easily identifiable features) to compare the 3D image 1230 with the stored reference image. As a result, the BIAT submodule 837 can perform identification, authentication, and / or tracking more efficiently.

[0086] Note that some captured images (e.g., blurred or filtered images) and image content (e.g., 3D images or recognized people, objects, or scenes) may be stored in Figure 1 , 2, 3A-3D, 400 and 500, and / or any of the electronic devices 100 and / or 200, and / or any of the electronic systems 300A-300D, 400 and / or 500 (e.g., in a trusted environment). The secure repository can be virtually and / or physically separated from the rest of the electronic system so that only applications and / or hardware residing within the trusted environment can access data stored in the secure repository. In some aspects, the secure repository can be formed at least in part within the memory 830. Therefore, certain captured images and image content can be stored within the secure repository of the memory 830 (e.g., within the image buffer 831).

[0087] In some embodiments, the neural network model 833 may reside at least partially in Figure 1 , 2 , 3A-3D, 400 and 500, and / or any of the electronic devices 100 and / or 200, and / or any of the electronic systems 300A-300D, 400 and / or 500. Placing the neural network model 833 within the trusted environment enables the neural network model 833 to perform machine learning (e.g., utilizing a CNN) on the protected captured images and image content.

[0088] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different techniques and technologies. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above specification may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0089] In addition, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative parts, blocks, modules, circuits and steps have been described above generally according to their functionality. Whether this functionality is implemented as hardware or software depends on specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application, but this implementation method decision should not be interpreted as causing a departure from the scope of the present disclosure.

[0090] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An example storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. In an alternative, the storage medium may be integrated into the processor.

[0091] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A device comprising: a display comprising a plurality of display pixels and a blank space, wherein the blank space separates each display pixel of the plurality of display pixels from the other display pixels; a first light source configured to emit light, the first light source being proximate to the display; a first camera disposed behind the display and comprising a plurality of first optical sensors aligned with the plurality of display pixels, the plurality of first optical sensors configured to detect reflections of the light emitted by the first light source transmitted through the empty space and to capture a first image based at least in part on the reflections, wherein the reflections are partially occluded by the plurality of display pixels; a second camera proximate the display, the second camera configured to capture a second image; as well as A depth map generator is configured to generate depth information about one or more objects in a field of view (FOV) of the first camera and the second camera based at least in part on the first image and the second image. 2 . The device of claim 1 , wherein the first light source is configured to emit near infrared (NIR) light, and each of the first and second cameras is configured to detect the NIR light. 3 . The device of claim 1 , wherein the first light source is configured to emit near infrared (NIR) light, the first camera is configured to detect the NIR light, and the second camera is configured to detect visible light.

4. The device of claim 1, wherein the display comprises an organic light emitting diode display.

5. The device of claim 1, wherein the display comprises a micro light emitting diode display.

6. The apparatus of claim 1, wherein each optical sensor of the plurality of first optical sensors is aligned with a display pixel of a region of the display, wherein the region includes a plurality of display pixels comprising a repeating pixel pattern.

7. The apparatus according to claim 1, further comprising: An image filter is configured to filter out noise or interference from the first image captured by the first camera.

8. The apparatus of claim 7, wherein the depth map generator is further configured to generate the depth information about the one or more objects in the FOVs of the first and second cameras based at least in part on the filtered first image.

9. The apparatus of claim 7, wherein the image filter filters out the noise or interference from the first image captured by the first camera using a neural network model.

10. The apparatus of claim 7, wherein the image filter filters out the noise or interference from the first image captured by the first camera using an inverse point spread function transform.

11. The apparatus of claim 1, wherein the depth map generator is further configured to generate a three-dimensional image based at least in part on the depth information.

12. A method comprising: receiving a first image captured by a first camera based at least in part on reflections of light emitted by a first light source transmitted through a blank space in an electronic display disposed in front of the first camera, wherein the reflections are partially occluded by a plurality of display pixels in the electronic display, wherein the electronic display includes the plurality of display pixels and the blank space, the blank space separating each display pixel of the plurality of display pixels from the other display pixels, and the first camera includes a plurality of first optical sensors aligned with the plurality of display pixels; receiving a second image captured by a second camera proximate to the electronic display; as well as Depth information about one or more objects in a field of view (FOV) of the first and second cameras is generated based at least in part on the first and second images.

13. The method according to claim 12, further comprising: Near infrared (NIR) light is emitted from the first light source, wherein each of the first camera and the second camera is configured to detect the NIR light.

14. The method according to claim 12, further comprising: Near infrared (NIR) light is emitted from the first light source, wherein the first camera is configured to detect the NIR light, and the second camera is configured to detect visible light.

15. The method according to claim 12, further comprising: Noise or interference is filtered out from the first image. 16 . The method of claim 15 , wherein generating the depth information about the one or more objects in the FOVs of the first and second cameras is also based at least in part on the filtered first image.

17. The method according to claim 12, further comprising: Noise or interference is filtered out from the first image using a neural network model.

18. The method according to claim 12, further comprising: Noise or interference is filtered out from the first image using an inverse point spread function transform.

19. The method according to claim 12, further comprising: A three-dimensional image is generated based at least in part on the depth information.

20. A processing system comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the processing system to: receiving a first image captured by a first camera based at least in part on reflections of light emitted by a first light source transmitted through a blank space in an electronic display disposed in front of the first camera, wherein the reflections are partially occluded by a plurality of display pixels in the electronic display, wherein the electronic display includes the plurality of display pixels and the blank space, the blank space separating each display pixel of the plurality of display pixels from the other display pixels, and the first camera includes a plurality of first optical sensors aligned with the plurality of display pixels; receiving a second image captured by a second camera proximate to the electronic display; as well as Depth information about one or more objects in a field of view (FOV) of the first camera and the second camera is generated based at least in part on the first image and the second image.

21. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 12 to 19.

Images