Camera light source

Abstract

A device includes a first camera, wherein the first camera includes one or more light sensors configured to detect visible light passing through a first surface in a first direction. The device also includes a light source configured to emit infrared (IR) light passing through the first surface in a second direction opposite to the first direction. The device also includes a second camera configured to detect reflections of the IR light.

Description

Technical Field

[0001] The present embodiments generally relate to light sources for cameras. Background Art

[0002] Some electronic devices, such as smartphones, tablet computers, laptops, and other computing systems, include cameras and displays. The camera is typically located in an area of ​​the device that provides an unobstructed view of the environment. Thus, when integrated on the same surface as the display, the camera may be located adjacent to the display, within a large, unsightly black border around the display, or within a notch or cutout. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] One innovative aspect of the subject matter of the present disclosure can be implemented in a device including a first camera, wherein the first camera includes one or more light sensors configured to detect visible light passing through a first surface in a first direction. The device may also include a light source configured to emit infrared (IR) light passing through the first surface in a second direction opposite to the first direction. The device may also include a second camera configured to detect reflections of the IR light.

[0005] Another innovative aspect of the subject matter of the present disclosure can be implemented in a camera assembly. The camera assembly may include a lens and one or more light sensors configured to detect visible light passing through the lens in a first direction. The camera assembly may also include a light source configured to emit light passing through the lens in a second direction opposite to the first direction.

[0006] Another innovative aspect of the subject matter of the present disclosure can be implemented in a camera assembly. The camera assembly can include one or more light sensors configured to detect visible light or IR light passing through a first surface of the camera assembly in a first direction. The camera assembly can also include a light source configured to emit light passing through the first surface in a second direction opposite to the first direction. The camera assembly can also include a light guide configured to steer light toward the first surface, wherein the first surface includes at least a portion of the light guide. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present embodiments are shown by way of example and are not intended to be limited by the figures in the accompanying drawings.

[0008] Figure 1 An example electronic system is shown in accordance with some embodiments.

[0009] Figure 2 An example camera assembly is shown in accordance with some embodiments.

[0010] Figure 3A An example camera assembly is shown in accordance with some embodiments.

[0011] Figure 3B An example camera assembly is shown in accordance with some embodiments.

[0012] Figure 3C An example camera assembly is shown in accordance with some embodiments.

[0013] Figure 3D An example camera assembly is shown in accordance with some embodiments.

[0014] Figure 3E An example camera assembly is shown in accordance with some embodiments.

[0015] Figure 3F An example camera assembly is shown in accordance with some embodiments.

[0016] Figure 3G An example camera assembly is shown in accordance with some embodiments.

[0017] Figures 4A-4K An example electronic system is shown in accordance with some embodiments.

[0018] Figure 5 A cross-section of an example electronic system is shown in accordance with some embodiments.

[0019] Figure 6 A cross-section of an example electronic system is shown in accordance with some embodiments.

[0020] Figure 7 A cross-section of an example electronic system is shown in accordance with some embodiments.

[0021] Figure 8 A cross-section of an example electronic system is shown in accordance with some embodiments.

[0022] Fig. 9 An example image captured by a camera disposed below a display is shown in accordance with some embodiments.

[0023] Fig.10 A block diagram of an example image processing system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0024] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" means being directly connected to or connected by one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific nomenclature is set forth to provide a thorough understanding of the various aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be needed to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the present disclosure. Some of the following detailed descriptions are presented according to the process, logic blocks, processing and other symbolic representations of the operation of the data bits in the computer memory. The interconnection between circuit elements or software blocks can be shown as a bus or a single signal line. Each of the bus may alternatively be a single signal line, and each of the single signal line may alternatively be a bus, and a single line or bus may represent any one or more of the countless physical or logical mechanisms for communication between components.

[0025] Unless otherwise expressly stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0026] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include packaging materials.

[0027] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0028] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors. The term "processor" as used herein may refer to any general purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0029] Figure 1 An example electronic system 100 is shown in accordance with some embodiments. The electronic system 100 may include a display 106 and cameras 102 and 104.

[0030] Display 106 may include a porous display, such as an organic light emitting diode (OLED) display or a micro light emitting diode (micro-LED) display, that contains holes or gaps between display pixels and / or display sub-pixels.

[0031] Each of cameras 102 and 104 may include an array of active pixel sensors or light sensors (e.g., photodiodes, complementary metal oxide semiconductor (CMOS) image sensor arrays, charge coupled device (CCD) arrays, and / or any other sensor capable of detecting wavelengths of light in the visible light spectrum, IR light spectrum (e.g., NIR or short wave IR), or ultraviolet (UV) light spectrum). In some embodiments, both cameras 102 and 104 may be configured to detect NIR light. In other embodiments, one of the two cameras 102 and 104 may be configured to detect visible light (e.g., red, green, and blue (RGB) color components of light) ("RGB camera"), and the other camera may be configured to detect NIR light. In addition, in some embodiments, each of cameras 102 and 104 may be configured to support depth sensing. For example, each of cameras 102 and 104 may be configured to capture images of an object or scene in the field of view (FOV) of the camera, the images may be combined, and depth information about the object or scene may be extracted from the combined images. The resolution of depth sensing may depend on the separation distance between cameras 102 and 104. In other words, the greater the separation distance, the greater the amount of depth information that can be derived from the images captured by cameras 102 and 104.

[0032] In some embodiments, one or both of the cameras 102 and 104 may be disposed behind the display 106. For example, when the camera 102 is disposed behind the display 106, the FOV of the camera may be partially blocked by the pixels and / or sub-pixels of the display 106. Therefore, when light passes through the display 106, some of the light may be blocked by the pixels and / or sub-pixels, and the remaining light may be detected by the camera 102. The camera 102 may capture the remaining light in an image, but the image may contain noise, distortion, or interference caused by the occlusion. In some embodiments, a neural network model may be used to filter out the noise or interference so that the image is suitable for depth sensing and / or other image processing. In addition to other advantages, disposing the camera 102 behind the display 106 may also eliminate the need for unsightly black borders, cutouts, or notches in the bezel of the display 106. Subsequently, the screen-to-body ratio of the electronic system 100 may be enhanced, and the electronic system 100 may support a borderless display.

[0033] In some embodiments, the under-display camera may require a light source in order to capture an image of an object or scene. Aspects of the present disclosure provide a camera assembly ( Figure 1The camera assembly may include a lens, a camera, and a light source. In some embodiments, the camera assembly may include a reflector configured to reflect light emitted from the light source through the lens. In other embodiments, the camera assembly may include a light guide configured to redirect light emitted by the light source through the first surface of the camera assembly.

[0034] Figure 2 An example camera assembly 200 is shown in accordance with some embodiments. Camera assembly 200 may include a light source 210, a reflector 216, a lens 212, a camera 214, a filter 218, a proximity sensor 254, and a housing 208.

[0035] A light source 210 is disposed within the housing 208, to the side of the lens 212. The light source 210 may include an LED, a vertical cavity surface emitting laser (VCSEL), a laser diode, or other light source capable of emitting wavelengths of light in the IR spectrum, including but not limited to NIR or short wave infrared (SWIR).

[0036] Reflector 216 is configured to reflect IR light emitted by light source 210 away from camera 214 and transmit visible light from lens 212 to camera 214. In some embodiments, reflector 216 may include a reflective plane, a dichroic reflector, and / or a dichroic mirror. In other embodiments, reflector 216 may include a beam splitter.

[0037] Lens 212 is an optical lens or lens assembly configured to transmit visible light and / or IR light. More specifically, lens 212 can be configured to transmit visible light 222 from outside of camera assembly 200 to camera 214 in an inward (or first) direction. In some aspects, lens 212 can be configured to focus visible light 222 onto camera 214. Lens 212 can also be configured to transmit IR light 220 emitted by light source 210 and reflected by reflector 216. Figure 2 As shown in , IR light 220 may travel in a direction opposite to the direction of visible light 222 , or in an outward (or second) direction, away from camera assembly 200 .

[0038] Camera 214 may include an array of active pixel sensors or light sensors (e.g., photodiodes, CMOS image sensor arrays, CCD arrays, and / or any other sensor capable of detecting wavelengths of light in the visible, IR, or UV spectrums). In some embodiments, camera 214 may be an RGB camera. Further, camera 214 may be configured to detect visible light 222 while light source 210 emits IR light 220. Further, camera 214 may be configured to perform wide range imaging, where objects or scenes are imaged at close or far distances.

[0039] The filter 218 is disposed between the camera 214 and the lens 212. The filter 218 may include an IR cut filter. The filter 218 may be configured to reflect and / or block IR light wavelengths (e.g., mid-IR wavelengths, NIR, or SWIR) so that the IR light does not interfere with the camera 214.

[0040] The proximity sensor 254 is disposed inside the housing 208, near the light source 210. The proximity sensor 254 may include an array of active pixel sensors or light sensors (e.g., photodiodes, CMOS image sensor arrays, CCD arrays, and / or any other sensor capable of detecting wavelengths of light in the visible spectrum, IR spectrum, or UV spectrum). In addition, the proximity sensor 254 may be configured to detect reflections of the IR light 220 emitted by the light source 210.

[0041] Housing 208 may include structure surrounding lens 212 and enclosing light source 210, reflector 216, filter 218, and camera 214. Figure 2 , first surface 258 is formed by the top of housing 208, the top of lens 212, and a portion of the side of lens 212. Additionally, length L1 is the length of housing 208 as measured from the top of housing 208 (at first surface 258) to the bottom of housing 208 (near camera 214). In some embodiments, it may be advantageous to minimize length L1 of housing 208.

[0042] In some embodiments, during operation, light source 210 may emit IR (e.g., NIR or SWIR) light 220 that reflects off reflector 216 and passes through lens 212 in an outward (or second) direction away from camera assembly 200. IR light 220 may illuminate a scene in front of camera assembly 200 and camera 214. Objects in the scene may reflect at least a portion of IR light 220 back toward camera assembly 200 and a display disposed near camera assembly 200. Some of the reflected IR light 220 may be blocked by pixels and / or sub-pixels in the display, while the remainder of the reflected IR light 220 passes through holes between display pixels and / or display sub-pixels. The reflected IR light 220 that passes through the holes may be detected by cameras ("IR cameras") configured to detect the reflected IR light 220. Each of the IR cameras may capture the detected IR light 220 in an image, and the captured image may be suitable for depth sensing.

[0043] In other embodiments, during operation, light source 210 can emit IR (e.g., NIR or SWIR) light 220 that reflects off reflector 216 and passes through lens 212 in an outward direction away from camera assembly 200. IR light 220 can illuminate a scene in front of camera assembly 200 and camera 214. Objects in the scene can reflect at least a portion of IR light 220 back toward camera assembly 200. Reflected IR light 220 can pass through lens 212, reflect off reflector 216, and be detected by proximity sensor 254. Thus, camera assembly 200 can be capable of emitting and sensing IR light to, for example, detect the presence of a user.

[0044] Figure 3A An example camera assembly 300A is shown in accordance with some embodiments. Camera assembly 300A may include a light source 310, a light guide 324, a housing 308, a lens 312, a filter 318, and a camera 314.

[0045] Light source 310 may be coupled to light guide 324 and configured to emit IR light 320. Light source 310 may include an LED, VCSEL, laser diode, or other light source capable of emitting wavelengths of light in the visible and / or IR spectrum, including but not limited to NIR or SWIR.

[0046] Light guide 324 may be configured to redirect IR light 320 emitted by light source 310 in an outward (or second) direction, away from camera assembly 300A. In some embodiments, light guide 324 may include a transparent material, such as transparent polycarbonate. In addition, light guide 324 may be configured to reflect light. For example, in some embodiments, light guide 324 may be configured to reflect light, wherein an inner surface of light guide 324 (which may include an outer surface of housing 308) has a refractive index greater than a refractive index of the inner surface of housing 308. As another example, light guide 324 may be configured to reflect light, wherein an inner surface of light guide 324 (which may include an outer surface of housing 308) includes a reflective surface, while the inner surface of housing 308 includes a dark surface or a surface capable of absorbing visible light or IR (e.g., NIR and / or SWIR) light. In some embodiments, light guide 324 may include (one or more) optical fibers, and in other embodiments, light guide 324 may include a mirror and / or a reflector. Furthermore, the light guide 324 may be coupled (eg, by means of molding) to the housing 308 .

[0047] Lens 312 is an optical lens or lens assembly that is configured to transmit visible light 322 from outside of camera assembly 300A in an inward (or first) direction to camera 314. In some aspects, lens 312 can be configured to focus visible light 322 onto camera 314. In some embodiments, lens 312 can include acrylic.

[0048] Camera 314 may include an array of active pixel sensors or light sensors (e.g., photodiodes, CMOS image sensor arrays, CCD arrays, and / or any other sensor capable of detecting wavelengths of light in the visible light spectrum, IR light spectrum (e.g., NIR or SWIR), or UV light spectrum). In some embodiments, camera 314 may include an RGB camera. Further, camera 314 may be configured to detect visible light 322 when light source 310 emits IR light 320. Further, camera 314 may be configured to perform wide range imaging, where objects or scenes are imaged at close or far distances.

[0049] Filter 318 is disposed between camera 314 and lens 312. Filter 318 may include an IR cut filter. Filter 318 may be configured to reflect and / or block IR light wavelengths (e.g., mid-IR wavelengths, NIR, or SWIR light) so that the IR light does not interfere with camera 314.

[0050] Housing 308 surrounds lens 312 and encloses filter 318 and camera 314. As described above, the interior surface of housing 308 (e.g., the wall facing camera 314) may include a dark surface or a surface configured to absorb visible light or IR (e.g., NIR and / or SWIR) light. Figure 3A, first surface 358 is formed by the top of housing 308, the top of lens 312, and the sides of light guide 324. Additionally, length L1 is the length of housing 308, as measured from the top of housing 308 (at first surface 358) to the bottom of housing 308 (near camera 314). In some embodiments, it may be advantageous to minimize length L1 of housing 308.

[0051] During operation, light source 310 can emit IR (e.g., NIR or SWIR) light 320, which is redirected through light guide 324 and first surface 358. IR light 320 can illuminate a scene in front of camera assembly 300A and camera 314. Objects in the scene can reflect at least a portion of IR light 320 back toward camera assembly 300A and a display disposed near camera assembly 300A. Some of the reflected IR light can pass through holes between pixels and / or sub-pixels of the display and be detected by an IR camera disposed behind the display. Each of the IR cameras can capture the detected IR light in an image, and the captured image can be suitable for depth sensing.

[0052] refer to Figure 3A , in some embodiments, light source 310 can be configured to emit IR (e.g., NIR or SWIR) light 320, and camera 314 can be configured to detect IR light 320 and / or visible light 322. Further, in some embodiments, light source 310 can be configured to emit IR light 320 while camera 314 detects IR light 320 and / or visible light 322. For example, during imaging, light source 310 can emit IR light 320 that is guided to the front of camera assembly 300A through light guide 324 and first surface 358. Some of IR light 320 can reflect off objects or features in front of camera assembly 300A and pass through lens 312 to camera 314. Thus, camera assembly 300A can be capable of emitting and sensing IR light for imaging.

[0053] In other embodiments, light source 310 can be configured to emit visible light. During imaging, light source 310 can emit visible light that is directed to the front of camera assembly 300A through light guide 324 and first surface 358. Some of the visible light can reflect off objects or features in front of camera assembly 300A and then pass through lens 312 to camera 314. Thus, camera assembly 300A can be able to emit and sense visible light for imaging.

[0054] Figure 3B An example camera assembly 300B is shown according to some embodiments. The camera assembly 300B may be the camera assembly 300B of the embodiment of the present invention. Figure 3A The embodiment of the camera assembly 300A described above. Figure 3B As shown in , camera assembly 300B includes two light sources 310, but in some embodiments, camera assembly 300B may include more than two light sources. Each light source 310 can be coupled to a light guide 324. In addition, each of light sources 310 can include an LED, a VCSEL, a laser diode, or other light source capable of emitting wavelengths of light in the IR spectrum (including but not limited to NIR or SWIR). During operation, each of light sources 310 can emit IR light 320, which travels through light guide 324 toward the front of camera assembly 300B. As described above with reference to Figure 2 and 3A As described, IR (eg, NIR or SWIR) light 320 may reflect off objects or features in front of camera assembly 300B and be detected by the under-display IR camera for depth sensing and image processing.

[0055] Figure 3C An example camera assembly 300C is shown according to some embodiments. The camera assembly 300C may be the camera assembly 300C of the embodiment shown in FIG. Figure 3A and 3B Embodiments of the camera assembly 300A and / or 300B described herein. Figure 3C As shown in , camera assembly 300C includes camera 314, which can be capable of detecting wavelengths of light in the visible spectrum or the IR spectrum (e.g., NIR or SWIR). In addition, camera assembly 300C includes light source 310, which can be capable of emitting wavelengths of light in the visible spectrum and / or the IR spectrum (e.g., NIR or SWIR).

[0056] like Figure 3C As shown in , the camera assembly 300C also includes a light guide 324. The light guide 324 includes a diffusing surface 326, which can be configured to diffuse, spread or scatter the light (e.g., visible light, NIR or SWIR) 320 emitted by the light source 310. The diffusing surface 326 can include a portion of the first surface 358, which forms a ring around the housing 308. In addition, the diffusing surface 326 can include a portion of the light guide 324. For example, the diffusing surface 326 can be integrated with the light guide 324 by means of molding. In some embodiments, the diffusing surface 326 may include a diffraction grating having a certain roughness. More specifically, the diffraction grating may include a large number of fine grooves, slits or ridges. In some embodiments, the grooves, slits and / or ridges can be randomly positioned, or they can be positioned in a parallel or uniform manner to form a pattern. Therefore, the diffusing surface 326 can be configured to provide a uniform illumination profile or a customized illumination pattern of the light 320.

[0057] Figure 3D An example camera assembly 300D is shown according to some embodiments. The camera assembly 300D may be the camera assembly 300D of FIG. Figure 3A , Figure 3B and Figure 3C Embodiments of camera assemblies 300A, 300B, and / or 300C are described. Figure 3D , the light guide 324 includes lenses and / or microlenses 328 (“microlenses 328”). In some embodiments, a plurality of microlenses 328 may be integrated (e.g., molded) on a portion of the first surface 358 forming a ring around the housing 308. The microlenses 328 may be configured to diffuse and / or diffract the IR light 320 emitted by the light source 310.

[0058] Figure 3E An example camera assembly 300E is shown according to some embodiments. The camera assembly 300E may be the camera assembly 300E of FIG. Figures 3A-3D Embodiments of camera assemblies 300A, 300B, 300C, and / or 300D are described. Figure 3E , the light guide 324 includes two micro lenses 330 disposed on either side of the housing 308. In some embodiments, more than two micro lenses 330 may be disposed on either side of the housing 308 along a surface of the light guide 324 forming a ring around the housing 308. The micro lenses 330 may be configured to diffuse and / or diffract light emitted by the light source 310.

[0059] Figure 3F An example camera assembly 300F is shown according to some embodiments. The camera assembly 300F may be the camera assembly 300F of FIG. Figures 3A-3E 300F. In some embodiments, camera assembly 300F may include ambient light sensor 352. Ambient light sensor 352 may be configured to detect an amount of ambient light in front of camera assembly 300F. During sensing, ambient light 350 in front of camera assembly 300F may be guided to ambient light sensor 352 via light guide 324. Thus, camera assembly 300F may be able to measure ambient light in order to, for example, adjust the brightness of a display screen coupled to camera assembly 300F.

[0060] Figure 3G An example camera assembly 300G is shown according to some embodiments. The camera assembly 300G may be the camera assembly 300G of the embodiment of the present invention. Figures 3A-3F Embodiments of camera assemblies 300A, 300B, 300C, 300D, 300E, and / or 300F are described.

[0061] like Figure 3G, camera assembly 300G includes a light guide 324 that can be configured to reflect light. In some embodiments, light guide 324 can be configured to reflect light, wherein an inner surface of light guide 324 has a refractive index greater than an inner surface of housing 308. Additionally, in some embodiments, light guide 324 can be configured to reflect light, wherein an inner surface of light guide 324 comprises a reflective surface, while an inner surface of housing 308 comprises a dark surface or a surface capable of absorbing visible light or IR (e.g., NIR and / or SWIR) light.

[0062] In some embodiments, camera assembly 300G may include proximity sensor 354. Proximity sensor 354 may be configured to detect IR (e.g., NIR or SWIR) light 320 emitted by light source 310. During sensing, light source 310 may emit IR light 320, which may be guided to the front of camera assembly 300G via light guide 324 (emitted IR light 320 is not shown for simplicity). Some of IR light 320 may reflect off objects or features in front of camera assembly 300G and travel back to proximity sensor 354 via light guide 324. Thus, camera assembly 300G may be able to emit and sense IR light to, for example, detect the presence of a user.

[0063] Figures 4A-4K Example electronic systems 400A-400K are shown in accordance with some embodiments. Each of electronic systems 400A-400K includes a camera assembly 432, cameras 402 and / or 404, and a display 406. Camera assembly 432 may be Figure 2 and Figures 3A-3G 4. The camera assemblies 200 and / or 300A-300G of FIG. 4 may each be Figure 1 Embodiments of cameras 102 and / or 104. In addition, each of cameras 402 and 404 can be disposed below display 406 and configured to capture images for depth sensing.

[0064] like Figure 4A As shown in FIG. 4 , camera assembly 432 is disposed near a corner of electronic system 400A, and cameras 402 and 404 are disposed along a long axis toward an end of electronic system 400A.

[0065] During operation, IR (e.g., NIR or SWIR) light can be emitted by the light source of camera assembly 432 to illuminate a scene in front of camera assembly 432. Objects in the scene can reflect at least a portion of the IR light back toward camera assembly 432 and display 406. Some of the reflected IR light can pass through holes between pixels and / or sub-pixels of display 406 and be detected by IR cameras 402 and 404 disposed behind display 406. Each of cameras 402 and 404 can capture the detected IR light in an image, and the captured image can be suitable for depth sensing.

[0066] exist Figure 4B , camera assembly 432 is disposed near a corner of electronic system 400B, and cameras 402 and 404 are disposed near a middle region of electronic system 400B along a long axis. Figure 1 As discussed, the separation distance between two cameras affects the resolution of depth sensing. In other words, the greater the distance between the cameras, the greater the amount of depth information that can be derived from the images captured by the cameras. Figure 4A , Figure 4B The cameras 402 and 404 are separated by a small distance. Therefore, Figure 4A The cameras 402 and 404 may support Figure 4B The cameras 402 and 404 have high depth sensing resolution.

[0067] Figure 4C Camera assembly 432 is shown disposed near a corner of electronic system 400C, and cameras 402 and 404 are disposed along the short axis toward a middle region of electronic system 400C. Figure 4D A camera assembly 432 disposed near a corner of the electronic system 400D is shown, as well as cameras 402 and 404 disposed in an upper portion of the electronic system 400D along a short axis.

[0068] Figure 4E A camera assembly 432 is shown disposed near one end of the electronic system 400E. Cameras 402 and 404 are disposed along the long axis, aligned with the camera assembly 432. Figure 4F In FIG. 4 , camera assembly 432 is disposed near one end of electronic system 400F. Cameras 402 and 404 are disposed along the long axis, aligned with camera assembly 432. Figure 4E , Figure 4F The cameras 402 and 404 are separated by a small distance. Therefore, Figure 4E Cameras 402 and 404 of FIG. 4 may support higher resolution depth sensing than cameras 402 and 404 of FIG. 5 .

[0069] Figure 4GA camera assembly 432 is shown disposed near one end of the electronic system 400G. The cameras 402 and 404 are disposed along a short axis toward a middle region of the electronic system 400G.

[0070] Figure 4H A camera assembly 432 is shown disposed near one end of the electronic system 400H and a camera 402 is shown disposed near an opposite end of the electronic system 400H. In some embodiments, both the camera assembly 432 and the camera 402 may be configured to capture images for depth sensing.

[0071] Fig. 4I A camera assembly 432 disposed near one end of the electronic system 400I and a camera 402 disposed near a middle region of the electronic system 400I are shown. In some embodiments, both the camera assembly 432 and the camera 402 may be configured to capture images for depth sensing.

[0072] Figure 4J A camera assembly 432 disposed near one corner of the electronic device 400J and a camera 402 disposed near an end of the electronic device 400J are shown. In some embodiments, both the camera assembly 432 and the camera 402 may be configured to capture images for depth sensing.

[0073] Figure 4K A camera assembly 432 disposed near one corner of the electronic system 400K and a camera 402 disposed near a middle region of the electronic system 400K are shown. In some embodiments, both the camera assembly 432 and the camera 402 may be configured to capture images for depth sensing.

[0074] Figure 5 5 shows a cross-section of an example electronic system 500 according to some embodiments. The electronic system 500 may be Figure 1 and Figures 4A-4G Embodiments of the electronic systems 100 and / or 400A-400G in FIG. Figure 5 As shown in FIG. 5 , electronic system 500 includes camera assembly 532, display layer 506, and module 540. Camera assembly 532 may be Figure 2 and Figures 4A-4G Module 540 includes cameras 502 and 504, each of which may be Figure 1 and Figures 4A-4G Embodiments of cameras 102, 104, 402 and / or 404.

[0075] The camera assembly 532 includes a light source 510, a reflector 516, a lens 512, a camera 514, a filter 518, and a housing 508. In addition, the length L1 is the length of the housing 508, as measured from the top of the housing 508 (at the first surface 558) to the bottom of the housing 508 (near the camera 514). In some embodiments, it may be advantageous to minimize the length L1 of the housing 508. In addition, although the length L1 appears to be shorter than the length L2 (which spans the length of the display layer 506 and the module 540), L1 can be almost any length that is shorter than, equal to, or longer than the length L2. In addition, although the camera assembly 532 is shown as being adjacent to the display layer 506 and the module 540, in actual implementations, components (not shown) may separate the camera assembly 532 from the display layer 506 and the module 540.

[0076] The display layer 506 may include multiple layers of a porous display such as an OLED display or a micro-LED display. Figure 5 As shown in , display layer 506 may include display surface 536 and emissive layer 538. In some embodiments, emissive layer 538 may include a plurality of display pixels and / or display sub-pixels with holes, gaps, or empty spaces between each display pixel and / or display sub-pixel.

[0077] Module 540 is disposed below display layer 506 and may include cameras 502 and 504 and corresponding lenses 542. Cameras 502 and 504 may include IR CMOS sensors configured for wide range imaging, where objects or scenes are imaged at close or far distances. Cameras 502 and 504 may also be configured to quickly sense objects such as fingers, hands, and heads. In addition, each of cameras 502 and 504 may be configured to capture images for depth sensing. The resolution of depth sensing may depend on the separation distance D between cameras 502 and 504. In other words, the greater the distance D, the greater the amount of depth information that can be derived from the images captured by cameras 502 and 504.

[0078] During operation, light source 510 can emit IR (e.g., NIR or SWIR) light 520 that reflects off reflector 516 and passes through lens 512. IR light 520 can illuminate camera assembly 532 and the scene in front of cameras 502 and 504. Objects in the scene can reflect at least a portion of IR light 520 back toward display layer 506, for example, as reflected light 520(R)(1) and 520(R)(2). Some of the reflected light 520(R)(1) and 520(R)(2) may be blocked by display pixels and / or display sub-pixels in emissive layer 538, while the remainder of the reflected light 520(R)(1) and 520(R)(2) passes through holes between display pixels and / or display sub-pixels. The reflected light 520(R)(1) and 520(R)(2) that passes through holes and lens 542 can be detected by cameras 502 and 504, respectively. Each of cameras 502 and 504 may capture their respective reflected IR light as a separate image, and these images may be combined for depth sensing.

[0079] Figure 6 600 is a cross-section diagram of an example electronic system 600 according to some embodiments. The electronic system 600 may be Figure 4H-4K Any of the embodiments of the electronic systems 400H-400K. Figure 6 As shown in FIG. 6 , electronic system 600 includes camera assembly 632, display layer 606, and module 640. Camera assembly 632 may be Figure 2 , 4A -4K and 5K camera assembly 200, 432 and / or 532 embodiments. The display layer 606 may be Figure 5 An embodiment of the display layer 506 in FIG. Figure 6 As shown in FIG. 6 , module 640 includes camera 604, which may be Figure 1 and Figures 4A-4K as well as Figure 5 Embodiments of cameras 102, 104, 402, 404, 502 and / or 504.

[0080] During operation, light source 610 may emit IR (e.g., NIR or SWIR) light 620 that reflects off reflector 616 and through lens 612. IR light 620 may illuminate a scene in front of camera assembly 632 and camera 604. Objects in the scene may reflect at least a portion of IR light 620 back toward display layer 606, for example, as reflected light 620(R). Some of reflected light 620(R) may pass through display layer 606 and lens 642, be detected by camera 604, and be captured as an image. In some embodiments, substantially simultaneously with light source 610 emitting IR light 620, camera 614 may detect visible light 622 (e.g., from the sun or another light source) that reflects off objects in the scene and capture visible light 622 in an image. Images captured by cameras 614 and 604 may be combined for depth sensing.

[0081] Figure 7 700 is a cross-section of an example electronic system 700 according to some embodiments. The electronic system 700 may be Figure 1 and Figures 4A-4G Embodiments of the electronic systems 100 and / or 400A-400G in FIG. Figure 7 As shown in FIG. 7 , electronic system 700 includes camera assembly 732, display layer 706, and module 740. Camera assembly 732 may be Figures 3A-3G The display layer 706 may be any of the embodiments of the camera assembly 300A and / or the camera assemblies 300B-300G. Figure 5 and Figure 6 In addition, module 740 may be an embodiment of display layer 506 and / or 606 in FIG. Figure 5 An embodiment of module 540.

[0082] The camera assembly 732 includes a light source 710, a light guide 724, a lens 712, a camera 714, a filter 718, and a housing 708. In some embodiments, the camera assembly 732 may also include additional light sources, diffusing surfaces, micro lenses, environmental sensors, and / or proximity sensors, such as Figure 3B-3G as shown in .

[0083] Length L1 is the length of housing 708, as measured from the top of housing 708 (at first surface 758) to the bottom of housing 708 (near camera 714). In some embodiments, it may be advantageous to minimize length L1 of housing 708. Furthermore, while length L1 appears shorter than length L2 (which spans the length of display layer 706 and module 740), L1 may be nearly any length shorter than, equal to, or longer than length L2. Additionally, while camera assembly 732 is shown adjacent to display layer 706 and module 740, in actual implementations, components (not shown) may separate camera assembly 732 from display layer 706 and module 740.

[0084] During operation, light source 710 can emit IR (e.g., NIR or SWIR) light 720, which is redirected through light guide 724 and first surface 758. IR light 720 can illuminate a scene in front of cameras 702, 704, and 714. Objects in the scene can reflect at least a portion of IR light 720 back toward display layer 706, for example, as reflected IR light 720(R)(1) and 720(R)(2). Some of the reflected IR light 720(R)(1) and 720(R)(2) can pass through display layer 706 and lens 742 and be detected by cameras 702 and 704, respectively. Each of cameras 702 and 704 can capture their respective IR light as separate images, and the images can be combined for depth sensing.

[0085] Figure 8 800 is a cross-section diagram of an example electronic system 800 according to some embodiments. The electronic system 800 may be Figure 4H-4K Any of the embodiments of the electronic systems 400H-400K. Figure 8 As shown in FIG. 8 , electronic system 800 includes camera assembly 832, display layer 806, and module 840. Camera assembly 832 may be Figures 3A-3G The display layer 806 may be any of the embodiments of the camera assembly 300A and / or the camera assemblies 300B-300G. Figure 5-7 In addition, module 840 includes a camera 804, which may be Figure 1 , Figures 4A-4K and Figure 5-7 804. In some embodiments, the camera 804 may be configured to detect IR (e.g., NIR or SWIR) light, and the filter 818 may be configured to shield the camera 804 from visible light.

[0086] During operation, light source 810 can emit IR (e.g., NIR or SWIR) light 820, which is redirected through light guide 824 and first surface 858. IR light 820 can illuminate a scene in front of cameras 804 and 814. Objects in the scene can reflect at least a portion of IR light 820 back toward camera assembly 800 and display layer 806, for example, as reflected IR light 820(R)(1) and 820(R)(2). Some of the reflected IR light 820(R)(1) can pass through display layer 806 and lens 842, be detected by camera 804, and be captured as an image. In addition, some of the reflected light 820(R)(2) can pass through lens 812 and filter 818 to camera 814, which can capture the reflected light 820(R)(2) in an image. Each of the images captured by cameras 804 and 814 can be combined for depth sensing.

[0087] In some embodiments, the camera 814 can be configured to detect visible light, and the filter 818 can be configured to shield the camera 814 from IR light. During operation, the light source 810 can emit IR (e.g., NIR or SWIR) light 820, which is redirected through the light guide 824 and the first surface 858. The IR light 820 can illuminate a scene in front of the cameras 804 and 814. Objects in the scene can reflect at least a portion of the IR light 820(R)(1) back toward the display layer 806, for example, as reflected light 820(R)(1). Some of the reflected light 820(R)(1) can pass through the display layer 806 and the lens 842, be detected by the camera 804, and be captured as an image. In some embodiments, substantially simultaneously with the light source 810 emitting the IR light 820, the camera 814 can detect visible light 822 (e.g., from the sun or another light source) that reflects off of objects in the scene and capture the visible light 822 in an image. Each of the images captured by cameras 814 and 804 may be combined for depth sensing.

[0088] Fig. 9 An example image 900 is shown captured by a camera disposed below a display according to some embodiments. Image 900 may be captured by a camera disposed behind or below a porous display, such as, for example, Figure 1 , Figures 4A-4K and Figure 5-8 Camera 102, 104, 402, 404, 502, 504, 604, 702, 704 or 804. Fig. 9, image 900 includes a grid of dark lines and bright spots - or a "screen door effect". The dark lines correspond to a pattern of display pixels and / or display sub-pixels and are produced by display pixels and / or display sub-pixels blocking some light from passing through the display. In contrast, the brighter spots correspond to gaps or holes between display pixels and / or display sub-pixels. The bright spots are produced by light that passes through the holes or gaps in the display to the camera below.

[0089] In some embodiments, the image captured by the under-display camera may include a grid or screen door effect, which appears as if the under-display camera is simultaneously imaging through multiple layers of screen doors (or a stack of screen doors), where each screen door layer has a different spacing (i.e., hole size or density). In such embodiments, light may pass through the holes or gaps between display pixels and / or display sub-pixels of the screen door layers. In addition, each light point spreads horizontally and vertically into different locations (i.e., as a point spread function), and each IR light point spreads across the image in a similar manner, which may cause significant distortion.

[0090] like Fig. 9 As shown in , portions of image 900 are obfuscated by pixels and / or sub-pixels of the display. In other words, display pixels and / or display sub-pixels can reduce the amount of light available for sensing and effectively block the FOV of the camera under the display. Therefore, in some embodiments, the captured image can be filtered or calibrated to eliminate screen door effects, distortion, or other artifacts.

[0091] Fig.10 1000 according to some embodiments. The image processing system 1000 includes a device interface 1010, a processor 1020, and a memory 1030. For the purposes of discussion herein, the processor 1020 is shown as being coupled to the device interface 1010 and the memory 1030. For actual embodiments, the device interface 1010, the processor 1020, and / or the memory 1030 may be connected together using one or more buses (not shown for simplicity). In some embodiments, the image processing system 1000 may be a processor 1020, a memory 1030, and a device interface 1010. Figure 1 , Figures 4A-4K and Figure 5-8 An application specific integrated circuit (ASIC) or other integrated circuit (IC) of the electronic system 100, 400A-400K and / or 500-800.

[0092] The device interface 1010 can transmit signals to and receive signals from input devices, including but not limited to one or more cameras, camera components and / or displays. In some embodiments, the device interface 1010 may include a display interface 1012. The display interface 1012 may be used to communicate with a display and / or provide a visual interface to a user of the image processing system 1000. In some embodiments, the device interface 1010 may also include an RGB camera interface 1014 and IR camera interfaces 1016 and 1018. Each camera interface 1014, 1016 and 1018 may be used to communicate with different cameras. For example, the RGB camera interface 1014 may transmit an activation signal to a camera 514 or 714 and receive sensor information from the camera 514 or 714 to capture an image of an object or scene. In addition, each of the IR camera interfaces 1016 and 1018 may transmit an activation signal to, for example, a camera 502, 504, 702 or 704 and receive sensor information therefrom to capture an IR image of an object and / or scene.

[0093] The memory 1030 may include an image buffer 1032 to store images received via the camera interfaces 1014, 1016, and / or 1018. The memory 1030 may also include a non-transitory computer-readable medium (e.g., one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, hard drive, etc.), which may store at least the following SW modules:

[0094] An image filtering SW module 1034 for filtering images received via the IR camera interface 1016 and / or 1018 (or other interfaces communicating with the camera under the display), the image filtering SW module 1034 also includes:

[0095] o a neural network model 1035 for filtering, reducing or removing noise from images received via IR camera interfaces 1016 and / or 1018; and

[0096] A depth map SW module (or depth map generator) 1036 for combining images received via (i) both IR camera interfaces 1016 and 1018 or (ii) RGB camera interface 1014 and IR camera interface 1016 or 1018 and extracting depth information from the images using, for example, stereo vision.

[0097] Each software module includes instructions that, when executed by processor 1020 , cause image processing system 1000 to perform a corresponding function.

[0098] For example, the processor 1020 may execute the image filtering SW module 1034 to filter images received via the IR camera interface 1016 and / or 1018. When executing the image filtering SW module 1034, the processor 1020 may use the neural network model 1035 to filter, reduce or eliminate noise (such as a screen door effect) from the images received via the IR camera interface 1016 and / or 1018. The processor 1020 may further execute the depth map SW module 1036 to determine the depth information of the objects or features in the filtered images. For example, the depth map generator may use stereo vision to combine the filtered images and extract the depth information from the combined, filtered images. The depth map generator may further generate a depth map of the object or feature based at least in part on the extracted depth information. In addition, the processor 1020 may also authenticate the subject, object or feature in the image based at least in part on the depth information.

[0099] As mentioned above Fig. 9 and Fig.10 As described, images captured by the under-display camera may include a screen door effect or other noise, distortion, or interference. Aspects of the present disclosure recognize that the screen door effect occurs when light (e.g., reflected IR, NIR, or SWIR light) passes through a display and some of the light is blocked by pixels and / or sub-pixels of the display, while the remaining light passes through holes between pixels and / or sub-pixels and is captured in an image by the under-display camera. To facilitate depth sensing or other applications, an image filter may use a neural network model to filter out the screen door effect from an image.

[0100] In some embodiments, the neural network model 1035 may include a CNN that is trained to filter out screen door effects or other noise from captured images. In other words, the CNN may be trained to infer an original or clean image of an object or scene from a blurred image of the object or scene. In some embodiments, the CNN may be trained by receiving and processing at least one original image of an object or scene and at least one blurred image of the object or scene. In some embodiments, an inverse point spread function transform may be used to filter images instead of or in addition to the CNN. Therefore, the image filtering SW module 1034 may use the neural network model 1035 and / or the inverse point spread function transform to filter images from an under-display IR camera.

[0101] Some captured images (e.g., blurred or filtered images) and image content (e.g., depth information, depth maps, or authenticated people, objects, or scenes) may be stored in Figure 1 , Figures 4A-4K and Figure 5-8The secure repository may be virtually and / or physically partitioned from the rest of the electronic system such that only applications and / or hardware residing within the trusted environment may access data stored in the secure repository. In some aspects, the secure repository may be formed at least in part within memory 1030. Thus, certain captured images and image content may be stored within the secure repository of memory 1030 (e.g., within image buffer 1032).

[0102] In some embodiments, the neural network model 1035 may reside at least partially in Figure 1 , Figures 4A-4K and Figure 5-8 Placing the neural network model 1035 in the trusted environment enables the neural network model 1035 to perform machine learning (e.g., using CNN) on the protected captured images and image content.

[0103] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0104] In addition, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0105] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An example storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor.

[0106] In the foregoing description, embodiments have been described with reference to specific examples of embodiments. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A device comprising: monitor; a first camera comprising one or more light sensors configured to detect visible light in a first direction passing through a first surface separate from the display; a light source configured to emit infrared (IR) light through the first surface in a second direction opposite to the first direction such that the IR light does not pass through the display in the second direction; a second camera disposed behind the display and configured to detect reflections of the IR light passing through the display in the first direction and to capture the reflections of the IR light in an image; as well as An image filter that uses a neural network model to filter out from the image a screen door effect caused by a portion of the reflected IR light being blocked by pixels and / or sub-pixels of the display. 2 . The device of claim 1 , wherein the first surface comprises a lens configured to focus the visible light onto the one or more light sensors.

3. The device according to claim 1, further comprising: A dichroic reflector is configured to transmit the visible light to the one or more light sensors while reflecting the IR light away from the one or more light sensors.

4. The apparatus according to claim 1, further comprising: A light guide is configured to divert the IR light toward the first surface, wherein the first surface comprises at least a portion of the light guide. 5 . The device of claim 4 , wherein the portion of the light guide comprises a diffusing surface configured to diffuse the IR light emitted from the first surface.

6. The device of claim 4, wherein the portion of the light guide comprises a plurality of microlenses configured to diffuse the IR light emitted from the first surface.

7. The device of claim 4, wherein the light guide is further configured to redirect ambient light emitted from the first surface to an ambient light sensor.

8. The device of claim 4, wherein the light guide is further configured to redirect reflections of the IR light emitted from the first surface to a proximity sensor.

9. The apparatus according to claim 1, further comprising: A depth map generator is configured to generate depth information about one or more objects in the field of view of the second camera based at least in part on the detected reflections of the IR light.

10. The device of claim 9, wherein the depth information is based on the detected reflections of the IR light and detected visible light.

11. The apparatus according to claim 9, further comprising: A third camera is configured to detect reflections of the IR light emitted by the light source, wherein the depth information is based on the reflections of the IR light detected by each of the second camera and the third camera.

12. The device of claim 11, wherein the third camera is arranged behind the display.

13. A device comprising: monitor; a lens separated from the display; one or more light sensors configured to detect visible light passing through the lens in a first direction; a light source configured to emit near infrared (NIR) light through the lens in a second direction opposite to the first direction such that the NIR light does not pass through the display in the second direction; a reflector configured to transmit the visible light to the one or more light sensors while reflecting the NIR light away from the one or more light sensors; a first camera disposed behind the display and configured to detect reflections of the NIR light passing through the display in the first direction and to capture the reflections of the NIR light in an image; as well as An image filter that uses a neural network model to filter out from the image a screen door effect caused by a portion of the reflected NIR light being blocked by pixels and / or sub-pixels of the display.

14. The device of claim 13, wherein the lens is configured to focus the visible light onto the one or more light sensors.

15. The apparatus of claim 13, wherein the reflector comprises a reflective surface.

16. The apparatus of claim 13, wherein the reflector comprises a dichroic reflector.

17. The apparatus of claim 13, further comprising: A proximity sensor is configured to detect a reflection of the NIR light emitted by the light source.

18. A device comprising: monitor; one or more light sensors configured to detect visible light in a first direction passing through a first surface separate from the display; a light source configured to emit light through the first surface in a second direction opposite to the first direction such that the emitted light does not pass through the display in the second direction; a light guide configured to turn the emitted light toward the first surface, wherein the first surface comprises at least a portion of the light guide; a first camera disposed behind the display and configured to detect reflections of IR light passing through the display in the first direction and to capture the reflections of the IR light in an image; as well as An image filter that uses a neural network model to filter out from the image a screen door effect caused by a portion of the reflected IR light being blocked by pixels and / or sub-pixels of the display.

19. The apparatus of claim 18, wherein the light emitted by the light source comprises near infrared (NIR) light, and wherein the portion of the light guide is configured to diffuse the NIR light emitted from the first surface.

20. The apparatus of claim 18, wherein the light emitted by the light source comprises the visible light, and wherein the portion of the light guide is configured to diffuse the visible light emitted from the first surface.

21. The apparatus of claim 18, wherein the light emitted by the light source comprises near infrared (NIR) light, and wherein the portion of the light guide comprises a diffuser or a plurality of microlenses configured to diffuse the NIR light emitted from the first surface.

22. The apparatus of claim 18, wherein the light emitted by the light source comprises the visible light, and wherein the portion of the light guide comprises a diffuser or a plurality of microlenses configured to diffuse the visible light emitted from the first surface.

Images