Sensor, system and method for compact facial recognition polarization camera

Abstract

A method for identifying an owner of a mobile device for unlocking the mobile device includes including in the mobile device a sparse color polarization (SCP) camera having a single SCP image sensor configurable and for capturing SPC images, wherein the SCP image sensor comprises a first plurality of N1 color pixel units and a second plurality of N2 polarization pixel units, wherein N1 > = 4N2; using the SCP camera to capture an SCP image of a user; analyzing the captured SCP image to determine whether the user is an owner of the mobile device; and unlocking the mobile device if the user is identified as the owner.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to and claims priority from U.S. Provisional Patent Application No. 63 / 482,847, filed on February 2, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments disclosed herein generally relate to biometric security methods, and more particularly, to unlocking a mobile device using biometric security methods. Background Art

[0004] It's common knowledge that mobile handheld electronic devices (hereinafter referred to as "mobile devices"), such as smartphones, tablets, headphones, smartwatches, and smart glasses, are equipped with biometric technology to unlock (or open) the mobile device. A popular biometric method is face unlock (or Face ID). This method uses facial image data of the mobile device owner to extract their unique (or characteristic) biometric features. Based on these biometric features, the mobile device unlocks when the owner approaches.

[0005] Figure 1A shows an example of a known mobile device 100 that is configured to support face unlocking. The mobile device 100 has a rear surface 102 that is generally facing away from the user. The mobile device 100 has a front surface 104 that is generally facing the user. The front surface 104 includes a first Selfie (or front-facing) camera 110 having a first self-timer camera field of view (FOV) FOV1 112. The first Selfie camera 110 can be a color (or C) camera as known in the art, for example, including a C image sensor, such as an RGB image sensor 150 (Figure 1C). In other examples, the C camera can include, for example, a C image sensor of red, green, blue, white (RGBW), cyan, magenta, yellow (CMY), red, blue, white (RBW), etc. Hereinafter, the first selfie camera 110 will be referred to as the "selfie C camera" 110. The front surface 104 may further include a second selfie camera 120 having a second selfie camera field of view FOV2 122. As shown, FOV1 112 and FOV2 122 overlap. In some examples, the second selfie camera 120 may be a depth camera known in the art for providing depth image data. In other examples, the second selfie camera 120 may be a C camera that captures C image data. The first selfie camera 110 and the second selfie camera 120 constitute a stereo camera that can be used to provide depth image data. Hereinafter, the second selfie camera 120 will be referred to as the "selfie depth camera" 120. The front surface 104 may include a screen 124. The rear surface 102 may include a multi-camera 126 known in the art. The sizes (or "sizes") of the various cameras are compatible with slim and compact mobile devices.

[0006] FIG1B illustrates a known face unlock method 130. In a first step 132, a user approaches a mobile device (e.g., a smartphone), such as mobile device 100 configured to perform face unlock. In a second step 134, the mobile device captures a selfie image (or a video stream of selfie images) containing a face (e.g., the user's face 114 in FIG1A ). In the first face unlock example, only the selfie C camera 110 is used to capture the selfie C image. This first face unlock example may be referred to as "single-camera face unlock." In a second face unlock example, the selfie C camera 110 and the selfie depth camera 120 are used to capture the selfie C image and the selfie depth image, respectively. This second face unlock example may be referred to as "dual-camera face unlock." In a third step 136, the mobile device analyzes the selfie C image to identify the owner of the mobile device. For dual-camera face unlock, in a fourth step 138, the mobile device analyzes the selfie depth image to identify the owner of the mobile device. In some dual-camera face unlock examples, only the selfie depth image may be used to identify the owner of the mobile device. In other dual-camera face unlocking examples, the self-portrait depth image can be fused (or "merged") with the self-portrait C-image to obtain a fused "C-depth" image. The C-depth image is then analyzed to identify the owner of the mobile device. To identify the owner of the mobile device, a set of facial features is typically extracted and compared with a set of facial features known to the owner (i.e., a set of facial features previously captured and stored by the mobile device). In the fifth step 140, if the comparison results in the extracted facial features being identical to the facial features known to the owner, the mobile device is unlocked. If the comparison results in the extracted facial features being significantly different from the facial features known to the owner, the mobile device is not unlocked.

[0007] The advantage of single-camera face unlock is that it can be implemented using relatively simple (and / or inexpensive) and compact camera hardware. The advantage of dual-camera face unlock over single-camera face unlock is that it provides higher security for the owner of the mobile device. Higher security means that the probability of dual-camera face unlocking incorrectly unlocking the mobile device is lower than that of single-camera face unlock. Falsely unlocking a mobile device means, for example, that the mobile device is unlocked even to someone who is not the owner of the mobile device, or that the mobile device is unlocked even though the owner's face is not in the scene (known as "face spoofing"). For face spoofing, the owner's image or a mask resembling the owner can be placed in FOV1 and FOV2. Higher security can be achieved by using additional image information. Here, the additional image information is depth image information. The disadvantage of dual-camera face unlock is that its implementation requires relatively complex (or expensive) and large camera hardware. Specifically, it requires two cameras (rather than one), which is disadvantageous in terms of cost, and it covers a larger area of ​​the mobile device screen, which is disadvantageous in terms of industrial design. Generally speaking, the security level of facial recognition is measured by statistical parameters, such as the Equal Error Rate (EER), True Positive Rate (TPR), or False Positive Rate (FPR), as known in the art, or their derivative parameters, such as the Receiver Operating Characteristic (ROC) curve and the Area under the ROC Curve (AUC). Therefore, a relatively low EER value (i.e., relatively close to zero) and a relatively high AUC value (i.e., relatively close to 1) represent a relatively high security level. Typically, security standards for mobile devices are defined by the Fast Identity Online (or "FIDO") Alliance.

[0008] FIG1C shows a top view of a known RGB image sensor 150. The RGB image sensor 150 is included in an RGB camera, and the optical axis of the RGB camera is perpendicular to the RGB image sensor 150. The RGB image sensor 150 includes a plurality of single pixels (red - R, green - G, and blue - B), such as a single R pixel 152. Typically, in a top view, a single pixel is square. The RGB image sensor 150 includes a plurality of pixel units, such as pixel unit 154. For example, pixel unit 154 includes one R pixel, one B pixel, and two G pixels. Pixel unit 154 represents the smallest pixel unit (or "component") that can be used as a building block for constructing ("assembling") the RGB image sensor 150. As shown in the figure, the RGB image sensor 150 can be formed by repeating pixel units (such as pixel unit 154), as shown in pixel units 156, 158, 160, and 162.

[0009] FIG1D shows a top view of a known polarization image sensor 170. The image sensor 170 is included in a polarization camera, the optical axis of which is perpendicular to the polarization image sensor 170. The polarization image sensor 170 includes a plurality of individual pixels, such as a single pixel 172. The polarization image sensor 170 includes a plurality of pixel units, such as a pixel unit 174. The pixel unit 174 includes a pixel that measures incident light with 90 degrees of linear polarization (labeled "90°"), a pixel that measures incident light with 45 degrees of linear polarization (labeled "45°"), a pixel that measures incident light with 135 degrees of linear polarization (labeled "135°"), and a pixel that measures incident light with 0 degrees of linear polarization (labeled "0°"). Typically, a polarization filter is applied to the top of each pixel. The pixel unit 174 represents the smallest pixel unit that can be used as a building block for constructing the polarization image sensor 170. As shown, the polarization image sensor 170 may be constructed by repeating pixel units (eg, pixel unit 174 ), as shown by pixel units 175 , 176 , 177 , and 178 .

[0010] Polarization image sensors, such as polarization image sensor 170, are known to be included in polarization cameras to provide additional image information. Here, the additional image information is polarization image information. Polarization image information can be used for dual-camera face unlocking (instead of depth image information) to provide a higher degree of security.

[0011] Facial features may represent characteristic shapes, distances between shapes, and the like presented in color and polarized images, respectively. For example, the contrast used to identify specific facial features in a color image may be based on color information, while the contrast used to identify specific facial features in a polarized image may be based on the degree of linear polarization (DOLP) or the angle of linear polarization (AOLP). The contrast in a polarized image may also be based on the degree of circular polarization (DOCP) or the angle of circular polarization (AOCP). For example, U.S. Patent No. 11,841,522 B2 describes an image sensor that can be used to measure DOCP and AOCP.

[0012] For example, it is well known that polarized image data can be used to determine the surface normal of the surface of the captured object. This fact can be used to prevent certain examples of face spoofing. Suppose in a face spoofing attack, the attacker uses a photo of the owner's face printed on a paper. In the C image, it is sometimes difficult (both for humans and algorithms) to distinguish between an image of the owner's real face and an image of the owner's face photo. However, by using, for example, surface normal information (which can be extracted from the polarized image, for example), it is possible to effectively distinguish between an image of the owner's real face (which exhibits a large variation in surface normal) and an image of the owner's face photo (which exhibits a small variation in surface normal). It is well known that screens (or "displays") usually emit polarized light. Therefore, polarized image data can be used to distinguish between a real face and a photo of the face displayed on the screen.

[0013] Therefore, it is necessary and beneficial to obtain a single-camera face unlock that can provide a higher degree of security than known single-camera face unlocks. Summary of the Invention

[0014] In various exemplary embodiments, a method is provided, comprising: providing a mobile device including a sparse color polarization (SCP) camera, the SCP camera including a single SCP image sensor configurable and operable to capture an SPC image, wherein the SCP image sensor includes a first plurality (N1) of color pixel units and a second plurality (N2) of polarization pixel units, where N1 ≥ 4N2; capturing an SCP image of a user; analyzing the captured SCP image to determine whether the user is an owner of the mobile device; and unlocking the mobile device if the user is identified as the owner.

[0015] In some embodiments, N1 ≥ 4N2. In some embodiments, N1 ≥ 16N2. In some embodiments, N1 ≥ 64N2. In some embodiments, N1 ≥ 128N2. In some embodiments, N1 ≥ 256N2. In some embodiments, N1 ≥ 1024N2. In some embodiments, N1 ≥ 4096N2.

[0016] In some embodiments, the second plurality of polarization pixel units are used to capture linear polarization information. In some embodiments, the second plurality of polarization pixel units are used to capture circular polarization information.

[0017] In some embodiments, the SCP image is used to create a color image and a polarization image.

[0018] In some embodiments, analyzing the captured SCP image includes analyzing the color image and analyzing the polarization image.

[0019] In some embodiments, the polarization image has a pixel resolution of at least 1200 pixels. In some embodiments, the polarization image has a pixel resolution of at least 4800 pixels. In some embodiments, the polarization image has a pixel resolution of at least 30,000 pixels.

[0020] In some embodiments, the mobile device has a front surface and a back surface, wherein the front surface includes a screen and the SCP camera is located on the front surface. In some embodiments, the back surface includes multiple cameras.

[0021] In some embodiments, the SCP camera includes a lens having an effective focal length (EFL) in the range of 2-10 mm. In some embodiments, the EFL is in the range of 3-6 mm.

[0022] In some embodiments, the single SCP image sensor has a full image sensor diagonal (SD) in the range of 3-17 mm. In some embodiments, the SD is in the range of 4-10 mm.

[0023] In some embodiments, the single SCP image sensor has a pixel resolution in the range of 5 megapixels (MP) to 400 MP. In some embodiments, the pixel resolution is in the range of 8 MP to 50 MP. In some embodiments, the pixel resolution is 12 MP. In some embodiments, the pixel resolution is 48 MP.

[0024] In some embodiments, the mobile device comprises a processor configured to perform said analyzing of the captured SCP image. In some embodiments, the mobile device comprises a memory for storing the first set of facial features of the owner. In some embodiments, the mobile device comprises a communication module configured to retrieve the first set of facial features of the owner.

[0025] In some embodiments, analyzing the captured SCP image includes calculating a second set of facial features and comparing the second set of facial features to the first set of facial features of the owner.

[0026] In various exemplary embodiments, a color polarization (CP) image sensor is provided, comprising: a first plurality (N1) of color pixel units, each color pixel unit including at least a first color pixel and a second color pixel; a second plurality (N2) of polarization pixel units, each polarization pixel unit including at least a first polarization pixel and a second polarization pixel; and a single metasurface layer, wherein the single metasurface layer covers light conversion regions of the first plurality of color pixel units and the second plurality of polarization pixel units; wherein the single metasurface layer is used to route light of a first color to the first color pixel and light of a second color to the second color pixel; and wherein the single metasurface layer is used to route light of a first polarization to the first polarization pixel and light of a second polarization to the second polarization pixel.

[0027] In some embodiments, the single metasurface layer comprises nanorods made of at least two different materials, wherein the at least two different materials have a lower refractive index n L and a higher refractive index n H .

[0028] In some embodiments, N1 ≥ 4N2. In some embodiments, N1 ≥ 16N2. In some embodiments, N1 ≥ 64N2. In some embodiments, N1 ≥ 128N2. In some embodiments, N1 ≥ 256N2. In some embodiments, N1 ≥ 1024N2. In some embodiments, N1 ≥ 4096N2.

[0029] In some embodiments, the CP image sensor is used to detect three different colors. In some embodiments, the three colors detected are red, green and blue.

[0030] In some embodiments, the CP image sensor is configured to detect two different polarization directions. In some embodiments, the CP image sensor is configured to detect four different polarization directions.

[0031] In some embodiments, the CP image sensor is configured to detect 0 degree linear polarization, 45 degree linear polarization, 90 degree linear polarization, and 135 degree linear polarization.

[0032] In some embodiments, the CP image sensor is configured to detect left-hand circular polarization and right-hand circular polarization.

[0033] In some embodiments, the height H of the single metasurface layer is in the range of 50 nm-5 μm. In some embodiments, the height H is in the range of 200 nm-2 μm.

[0034] In some embodiments, the lower refractive index n L In the range of 1-2.5, and the higher refractive index n H In the range of 1.25-5. In some embodiments, the lower refractive index n L In the range of 1-1.75, and the higher refractive index n H In the range of 2-4.

[0035] In some embodiments, the first plurality of color pixel units include a plurality of color pixels and the second plurality of polarization pixel units include a plurality of polarization pixels, the plurality of color pixels and the plurality of polarization pixels have the same pixel pitch P, and the pixel pitch P is in the range of 0.25-5 μm. In some embodiments, the pixel pitch P is in the range of 0.35-2 μm.

[0036] In some embodiments, the CP image sensor has a pixel resolution ranging from 5 megapixels (MP) to 400 MP. In some embodiments, the pixel resolution is in the range of 8 MP to 50 MP. In some embodiments, the pixel resolution is 12 MP. In some embodiments, the pixel resolution is 48 MP.

[0037] In some embodiments, the CP image sensor is included in a camera comprising a lens having an effective focal length (EFL) in the range of 2 mm to 10 mm. In some embodiments, the EFL is in the range of 3-6 mm.

[0038] In some embodiments, the CP image sensor has a full image sensor diagonal (SD) in the range of 3-17 mm. In some embodiments, the SD is in the range of 4-10 mm.

[0039] In some embodiments, the CP image sensor is included in a mobile device. In some embodiments, the mobile device has a front surface including a screen, and the CP image sensor is included in a camera located on the front surface.

[0040] In some embodiments, the mobile device is a smartphone. In some embodiments, the mobile device is a tablet computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The following description will provide a non-limiting example of the embodiments disclosed herein with reference to the accompanying drawings listed below. The drawings and descriptions are intended to illustrate and explain the embodiments disclosed herein and should not be construed as limiting the same in any way. Identical elements in different drawings may be represented by the same numerals.

[0042] Elements in the drawings are not necessarily drawn to scale. In the drawings:

[0043] FIG1A schematically illustrates a mobile device configured to perform face unlocking as known in the art;

[0044] FIG1B shows a method of performing face unlocking known in the art;

[0045] FIG1C shows an RGB image sensor known in the art;

[0046] FIG1D shows a polarization image sensor known in the art;

[0047] Figure 1E A sparse color polarization (SCP) image sensor known in the art is shown;

[0048] Figure 2A An example of an SCP selfie image for performing single-camera face unlocking disclosed herein is shown;

[0049] Figure 2B An example of an SCP image sensor disclosed herein is shown in top view;

[0050] Figure 2C Shown in cross-sectional side view Figure 2B The SCP image sensor shown;

[0051] Figure 2D Shown in cross-sectional side view Figure 2B Metasurface layers included in the SCP image sensor shown;

[0052] Figure 2E Shown in top view Figure 2D Portions of the supersurface layer shown;

[0053] Figure 2F Shown in top view Figure 2D Other parts of the supersurface layer are shown;

[0054] Figure 3 The single-camera face unlocking method disclosed herein is shown;

[0055] Figure 4 An embodiment of a mobile device configured to perform single-camera face unlocking disclosed herein is schematically illustrated. DETAILED DESCRIPTION

[0056] Figure 1E A known sparse color polarization (SCP) image sensor 180 is shown. The SCP image sensor 180 includes a single pixel, such as a single pixel 182, arranged in a first plurality of color pixel cells 184 (similar to cells 154) and a second plurality of polarization pixel cells 186 (similar to cells 174). For example, as shown, the color pixel cells 184 are RGB pixel cells. The polarization pixel cells 186 are "sparsely" integrated between the color pixel cells 184. "Sparsely integrated" herein means that the SCP image sensor 180 includes more color pixel cells 184 than polarization pixel cells 186. As shown, for example, the color pixel cells 184 are located at position 188 and the polarization pixel cells 186 are located at position 190. In general, the SCP image sensor 180 can include N independent pixels, "independent" referring to each R, G, B pixel, or 0°, 45°, 90°, or 135° pixel. Among the N independent pixels of the SCP image sensor, the proportion "r" of polarization pixels can be r≤1 / 2 i , i = 2, 3, 4, etc. Therefore, the SCP image sensor 180 includes N×r single polarization pixels and N×(1-r) single color pixels. Figure 1E In the example of , r=1 / 4. In other examples, r may be r≤1 / 16 or r≤1 / 64, etc. Hereinafter, r will be referred to as the "sparseness" of the polarization pixel.

[0057] In other examples (not shown), polarization pixel units (e.g., unit 186) can also be configured to capture color image data. For example, in addition to polarization filters, such polarization pixel units can also cover color filters. In such examples, a single pixel (not shown) can simultaneously measure polarization signals and color signals. That is, a single pixel can measure one polarization direction and one color.

[0058] Figure 2AAn example of an SCP selfie image 200 for performing single-camera face unlocking disclosed herein is shown. The SCP selfie image 200 is captured by a sparse CP selfie camera, which includes an SCP image sensor such as the SCP image sensor 180. The SCP selfie camera can be a front-facing camera of a mobile device. The captured SCP selfie image 200 may include the face of the selfie user. The SCP selfie image 200 includes color (e.g., RGB) image data 202 captured by a color pixel unit (e.g., unit 184) and polarization image data 204 captured by a polarization pixel unit (e.g., unit 186). By way of example, the polarization image data represented by rectangle 204 is only available at 9 locations in the SCP selfie image 200. Typically, the number of such locations will be much larger, for example, at least hundreds, thousands, or even hundreds of thousands of such locations in the SCP image.

[0059] It is worth noting that for each image pixel (or image point) in the SCP selfie image 200, only color image data or only polarization image data is available. It should be noted that in dark (or "low light") scenes, polarization image data can generally provide stronger contrast than color image data. For example, DOLP image data or AOLD image data has stronger contrast than color image data. It is well known that single-camera face unlocking performs significantly worse in dark scenes than dual-camera face unlocking. This is because for dual-camera face unlocking, the scene is typically artificially illuminated, such as the near-infrared (NIR) spectrum, which is invisible to humans. Typically, artificial lighting is not possible or required for single-camera face unlocking scenarios. Therefore, the higher the contrast of the polarization image data, the more conducive it is to performing single-camera face unlocking.

[0060] Now turning to the processing of the captured image data. After performing RGB demosaicing processing known in the art on the SCP selfie image 200, each pixel in the RGB image data 202 area can obtain three values ​​(one each for R, G, and B). After performing polarization demosaicing processing known in the art on the SCP selfie image 200, each pixel in the polarization image data 204 area can obtain four values ​​(one each for 90°, 45°, 135°, and 0°). In some examples, a pixel cell such as the polarization pixel cell 186 can be treated as a single "larger" pixel. For this pixel cell containing a larger pixel, four values ​​(one each for 90°, 45°, 135°, and 0°) can be obtained after performing polarization demosaicing processing. It is worth noting that two separate images can be created from the SCP selfie image 200, for example, a polarization image containing only polarization image data and a color image containing only color image data.

[0061] When using the SCP selfie camera to take a photo, that is, to capture an output image that is output (or "displayed") to the user, only color pixel units are typically used, and polarization pixel units are not used. For taking photos, relatively high sparsity is beneficial for achieving relatively high image quality (IQ) because more color pixel units contribute to the output image. For performing the single-camera face unlocking ( Figure 3 ), relatively high sparsity (e.g., N ≥ 64, N ≥ 256, N ≥ 1024, or even N ≥ 4096) may be sufficient to achieve relatively high security.

[0062] A CP image sensor, such as the SCP image sensor 180, the SCP selfie image sensor 220, or the CP image sensor 225 (see below), may have a pixel (or "spatial") resolution in the range of 2.5 megapixels (MP) to 400 MP, typically in the range of 8 MP to 50 MP, such as 12 MP or 48 MP. The aspect ratio of the SCP image sensor may be 4:3 or 16:9. The pixel resolution of the polarization image containing only polarization image data may be in the range of about 500 pixels to 15 MP. For example, the pixel resolution may be about 40×30 (1200 pixels), about 80×60 (4800 pixels), or about 200×150 (30,000 pixels), etc.

[0063] In terms of camera and image sensor parameters, a camera including a CP or SCP image sensor may include a lens with an EFL in the range of 2mm-15mm or 3mm-6mm, and the CP or SCP image sensor may have a (full) image sensor diagonal SD in the range of 2mm-17mm or 4mm-10mm.

[0064] Figure 2B FIG2 shows a top view of an example of an SCP Selfie image sensor disclosed herein (SCP Selfie image sensor 220). The SCP Selfie image sensor 220 includes a color image sensor region 222 containing color pixel units as disclosed herein and a polarization image sensor region 224 containing polarization pixel units as disclosed herein.

[0065] Figure 2CA cross-sectional side view of an exemplary color-polarization image sensor 225 disclosed herein is shown. The cross-section shows a first color pixel unit 226, a polarization pixel unit 228, and a second color pixel unit 230. Along the z-axis in the zx coordinate system shown, the pixel unit includes a metasurface layer 232, an (optional) spacer layer 234, an (optional) filter layer 236, and a pixel layer 238. Both the color image sensor region 222 and the polarization image sensor region 224 are covered by the metasurface layer 232.

[0066] Along the x-axis, the metasurface layer 232 includes a first color router 240 in the first color pixel unit 226 , a polarization router 242 in the polarization pixel unit 228 , and a second color router 244 in the second color pixel unit 230 . The first color router 240 and the second color router 244 can be "metasurface color-routers" (or "metasurface nano-prisms") known in the art. Metasurface color routers can be used to route colors to specific pixel areas or into specific pixels. For example, Chulsoo Choi et al. described metasurface color routers in a paper titled "Optical design of dispersive metasurface nano-prism structure for high sensitivity CMOS image" presented at the IEEE International Electron Devices Meeting (IEDM) held from December 9 to 13, 2023. Metasurface color routers can be used instead of color filters or in combination with color filters, which is beneficial because the CP selfie image sensor 225 with metasurface color filters can collect more light signals compared to a color image sensor area using only color filters.

[0067] Metasurface lenses (or "metalenses") that route specific polarization directions to specific image sensor regions are well known, and are used, for example, in the Metalenz product "Polar ID." The polarization router 242 may include "metasurface polarization-routers" (or simply "polarization-routers"), which are polarization-sensitive metalenses that are used to route specific polarization directions to specific pixel regions or specific pixels. Polarization routers can be used in place of polarization filters or in combination with polarization filters. The advantage of this is that the SCP selfie image sensor 220 with polarization routers can capture more light signals than a polarization image sensor region using only polarization filters. Both color routers and polarization routers can be referred to as "nano-routers."

[0068] In some examples, CP image sensor 225 can be a sparse CP image sensor, e.g., similar to SCP image sensor 220. In these examples, CP image sensor 225 can include a first plurality (N1) of color pixel cells and a second plurality (N2) of polarization pixel cells, where N1>N2.

[0069] In other examples, CP image sensor 225 may not be a sparse CP image sensor. In these examples, CP image sensor 225 may include a first plurality (N1) of color pixel cells and a second plurality (N2) of polarization pixel cells, where N1=N2.

[0070] In some examples, the CP image sensor 225 can be included in a camera located on a front surface (e.g., front surface 104). In other examples, the CP image sensor 225 can be included in a camera located on a rear surface (e.g., rear surface 102) of the mobile device. Generally speaking, a CP image sensor (e.g., SCP image sensor 180, SCP image sensor 220, or CP image sensor 225) can be used to collect visible light. In some examples, the SCP image sensor can be used to collect light in the near-infrared (NIR) range, i.e., light with a wavelength range of approximately 750 nm to 1000 nm. In other cases, the CP image sensor can be used to collect light in the short-wave infrared (SWIR) range, i.e., light with a wavelength range of approximately 1000 nm to 1800 nm. In other examples, the CP image sensor can be used to collect visible light, infrared light, and SWIR light, i.e., light with a wavelength range of approximately 450 nm to 1800 nm. In all of these examples, "color" can refer to specific wavelength bands within the visible, NIR, and SWIR wavelength regions, respectively.

[0071] Optional spacers 234 may be used to provide adequate spatial separation of the spectral and polarization components of light, respectively. In some examples, optional filters 236 may be known color filters for compensating for non-ideal spatial separation of the spectral components of light. In other examples, optional filters 236 may be known polarization filters for compensating for non-ideal spatial separation of the polarization components of light. In other examples, optional filters 236 may be color filters in color pixel cells or polarization filters in polarization pixel cells. As is well known, photon-to-electron conversion (or "photon collection") occurs in pixel layer 238.

[0072] Figure 2CThe diagram schematically illustrates green light (G) and red light (R) incident on the CP image sensor 225 from a direction parallel to the z-axis. Illustratively, the first color pixel unit 226 and the second color pixel unit 230 include G pixels 246 and R pixels 248, and G pixels 254 and R pixels 256, respectively. Illustratively, the polarization pixel unit 228 includes a 90-degree linear polarization ("90°") pixel 250 and a 45-degree linear polarization ("45°") pixel 252. From the first color router 240 and the second color router 244, respectively, any G light (solid lines) impinging on the first color pixel unit 226 and the second color pixel unit 230 is routed (or "guided") to the G pixels 246 and 254, respectively, while any R light (dashed lines) impinging on the first color pixel unit 226 and the second color pixel unit 230 is routed to the R pixels 248 and 256, respectively. Starting from the polarization router 242, any 90-degree linearly polarized light (solid line) impinging on the polarization pixel unit 228 is routed to the 90-degree pixel 250, and any 45-degree linearly polarized light (dashed line) impinging on the polarization pixel unit 228 is routed to the 45-degree pixel 252. The pixel pitch (or "pixel width") of a single pixel is represented by "P", and P can be in the range of 0.25 μm-5 μm, preferably in the range of 0.35 μm-2 μm, for example, 0.5 μm, 0.6 μm, or 0.7 μm, etc.

[0073] Figure 2D The cross-sectional side view of the metasurface layer 232 disclosed herein is exemplarily shown, which can be used in a CP image sensor similar to the CP image sensor 225. Here, exemplarily, the metasurface layer 232 has a symmetry axis parallel to the x-axis. Typically, the metasurface layer 232 is created (or manufactured) by forming nanostructures (also called "nanopillars" or "nanorods") made of two or more different materials. For example, the first type of nanostructure 258 can be made of a relatively low refractive index ("n L ”) and the second type of nanostructures 260 may be made of a first material having a relatively high refractive index (“n H ”) is made of the second material. Generally speaking, n H >n L .n L It can be in the range of 1 to 2.5, preferably in the range of 1-1.75, n H It may be in the range of 1.25 to 5, preferably in the range of 2 to 4 or 2 to a larger value. In some examples, n L It can be air. Figure 2D, each white region of the supersurface layer 232 represents a first type of nanostructure 258, and each black region of the supersurface layer 232 represents a second type of nanostructure 260. Two regions are shown, each region containing two types of nanostructures 258 and 260. The height ("H") of the supersurface layer 232 (and the nanostructures) can be in the range of 50nm-5μm, preferably in the range of 200nm-2μm. The width ("W") of a single nanostructure can be in the range of 20nm-1μm, preferably in the range of 50nm-500nm.

[0074] In some examples, as shown, color routers 240 and 244 and polarization router 242 are all made (or "fabricated") from the same two or more different materials. In other words, there are some material combinations that can be used to fabricate both color routers and polarization routers, thereby enabling the implementation of both color routers and polarization routers in a single metasurface layer. The functionality of color routing and polarization routing is primarily achieved through the geometry of the nanostructures ( Figure 2E ) and the interactions between the nanopillars, rather than necessarily through a combination of materials. This means that a "single" or "continuous" metasurface layer (e.g., metasurface layer 232) can provide color routing and polarization routing functions. In terms of manufacturing time, process complexity, and manufacturing cost, this is very advantageous for manufacturing the CP image sensor 225 disclosed herein because it allows the color routers 240 and 244 and the polarization router 242 to be manufactured in one or more of the same process steps required to manufacture a single metasurface layer. That is, the same process steps can be used to manufacture the color routers 240 and 244 and the polarization router 242. Here, the metasurface layer 232 uses a single layer of nanostructures. In other examples, the metasurface layer (e.g., metasurface layer 232) can include two or more layers of nanostructures, i.e., a nanostructure stack. For these other examples, if the color routing and polarization routing functions are introduced in the same process step, especially when the same two or more different materials are used, it can be referred to as a "single" or "continuous" metasurface layer.

[0075] Figure 2E The top view of the color router 240 is exemplarily shown. As an example, the color router 240 can spatially separate light into three different colors: R, G, and B.

[0076] Figure 2F The top view of the polarization router 242 is exemplarily shown. The polarization router 242 can spatially separate light into four different linear polarization directions, namely 90°, 45°, 0° and 135°.

[0077] As shown in the figure, the geometric shapes (or "shapes") of the nanostructures included in color router 240 and polarization router 242 exhibit structural differences. Specifically, the nanostructures included in color router 240 may primarily include geometric shapes that are rotationally symmetric along the z-axis. The nanostructures included in polarization router 242 may primarily include geometric shapes that are not rotationally symmetric along the z-axis but have a preferred directionality in the xy plane. "Primarily" here means that the proportion of rotationally symmetric geometric shapes may be greater in the color router than in the polarization router, and vice versa, the proportion of geometric shapes with a preferred directionality may be greater in the polarization router than in the color router. However, the color router may also include geometric shapes with a preferred directionality, and the polarization router may also include geometric shapes with a rotationally symmetric directionality.

[0078] Figure 3 The single-camera face unlock method 300 disclosed herein is shown. In step 302, a user approaches a mobile device configured to perform single-camera face unlock. In step 304, the mobile device captures an SCP self-portrait image 200 (or a stream of SCP self-portrait images) containing the user's face (e.g., face 114). In step 306, the mobile device analyzes a color image created from the SCP self-portrait image 200 to identify the face of the mobile device owner. In step 308, the mobile device analyzes a polarization image created from the SCP self-portrait image 200 to identify the face of the mobile device owner. In steps 306 and 308, a set of facial features can be extracted from the SCP self-portrait image 200 and compared with the facial features of the owner. Typically, the facial features extracted in steps 306 and 308 are different, which facilitates relatively high security. In other examples, facial spoofing detection can be performed in step 308. Facial spoofing detection can include analyzing the polarization image to distinguish a real face from a face photo displayed on a screen or printed on paper.

[0079] In step 310, if the mobile device finds that the set of facial features extracted in steps 306 and 308 are the same as the facial features of the mobile device owner, the mobile device is unlocked. If the mobile device finds that the set of facial features extracted in steps 306 and 308 are different from the facial features of the mobile device owner, the mobile device is not unlocked.

[0080] In the example of single-camera face unlocking, the mobile device can analyze the SCP self-portrait image 200 as it is by performing steps 306 and 308 in the same step. In other words, the color image data and the polarization image data can be analyzed together in one step.

[0081] In some examples of single-camera face unlocking, the mobile device creates a fused CP image in a first sub-step by performing steps 306 and 308 in the same step. In some of these examples, color image data is used to estimate (or artificially generate) polarization image data that does not actually exist in the SCP selfie image 200. In other examples, polarization image data is used to estimate color image data that does not actually exist in the SCP selfie image 200, and then a fused CP image is created. In one example, each pixel of the fused CP image may have seven values ​​(one each for R, G, B, 90°, 45°, 135°, and 0°). The latter can be called a "dense CP image." In a second sub-step, the mobile device analyzes the created image to recognize the face of the owner of the mobile device.

[0082] In some single-camera face unlocking examples, as a first sub-step of step 306, image processing methods known in the art (such as "super-resolution" or "upsampling") can be used to create a "high-resolution color image," i.e., a color image with a higher pixel resolution. For example, the pixel resolution of the created color image can be the same as the pixel resolution of the SCP selfie image. In a second sub-step of step 306, the mobile device analyzes the high-resolution color image to recognize the face of the mobile device owner.

[0083] In some single-camera face unlock examples, as a first sub-step of step 308, an image processing method can be used to create a "high-resolution polarization image," i.e., a polarization image with high pixel resolution. For example, the pixel resolution of the polarization image created can be the same as the pixel resolution of the SCP selfie image. In a second sub-step of step 308, the mobile device analyzes the high-resolution polarization image to identify the face of the mobile device owner.

[0084] The advantage of the method 300 over known single-camera face unlocking methods is that it provides higher security. The higher security is achieved by using additional image information. Specifically, the additional image information here is polarization image information.

[0085] The advantage of the method 300 over known dual-camera face unlocking methods is that it can be implemented using relatively simple (or inexpensive) and compact camera hardware. Specifically, it only requires one camera (instead of two), which is advantageous in terms of cost, and it occupies a smaller screen area (e.g., screen 124), which is advantageous in terms of industrial design.

[0086] To perform the method 300 , a mobile device, such as the mobile device 400 , is provided. Figure 4Schematically illustrates an embodiment of a mobile device 400 (e.g., a smartphone) configured to perform single-camera face unlocking as disclosed herein. The mobile device 400 includes a selfie camera 410 having a selfie camera FOV S The mobile device 400 also includes an image sensor 412. Image sensor 412 can be a sparse CP image sensor, such as SCP image sensor 180, SCP image sensor 220, or CP image sensor 225 as described above. Optionally, mobile device 400 can also include a second camera, such as a multi-camera as known in the art. In other examples, the CP image sensor can be integrated into the multi-camera.

[0087] Mobile device 400 also includes an application processor (AP) 420. AP 420 includes: a face detector 422 (e.g., configured to perform facial detection); a color face recognizer 424 (e.g., configured to recognize the face of the owner of mobile device 400 in a color image); a polarization face recognizer 426 (e.g., configured to recognize the face of the owner of mobile device 400 in a polarization image and / or detect facial spoofing); and an image sensor controller 428. Mobile device 400 also includes a screen 430 and a memory 440. Memory 440 can be used to store a set of facial features of the owner of mobile device 400. This stored set of facial features can be used for single-camera face unlocking in steps 306 and / or 308. In other examples, mobile device 400 can also include a communication module for retrieving a set of facial features of the owner of mobile device 400. For example, the communication module can retrieve the set of facial features of the owner of mobile device 400 from the internet, a server, a cloud, and so on.

[0088] Although the present disclosure has been described by certain embodiments and related methods thereof, it will be understood by those skilled in the art that variations and permutations of these embodiments and methods are obvious. The present disclosure should be understood not to be limited to the specific embodiments described herein, but only to the scope of the appended claims.

[0089] All references mentioned in this specification are incorporated herein by reference in their entirety, to the same extent as if each reference was specifically and individually indicated to be incorporated herein by reference. In addition, the citation or identification of any reference in this application should not be understood as an admission that the reference is available as prior art relative to the present application.

Claims

1. A method comprising: Providing a mobile device including a sparse color polarization (SCP) camera, the SCP camera including a single SCP image sensor configurable and operable to capture an SPC image, wherein the SCP image sensor includes a first plurality (N1) of color pixel units and a second plurality (N2) of polarization pixel units, wherein N1 ≥ 4N2; Capture an image of the user's SCP; analyzing the captured SCP image to determine whether the user is the owner of the mobile device; as well as If the user is identified as the owner, the mobile device is unlocked. The method according to claim 1 , wherein N1 ≥ 16N2. The method according to claim 1 , wherein N1 ≥ 64N2. The method according to claim 1 , wherein N1 ≥ 128N2. The method according to claim 1 , wherein N 1 ≥ 256 N 2 . The method according to claim 1 , wherein N1 ≥ 1024 N2 . The method according to claim 1 , wherein N1 ≥ 4096N2. The method of claim 1 , wherein the second plurality of polarization pixel units are used to capture linear polarization information.

9. The method of claim 1, wherein the second plurality of polarization pixel units are used to capture circular polarization information.

10. The method of claim 1, wherein the SCP image is used to create a color image and a polarization image.

11. The method of claim 10, wherein said analyzing the captured SCP image comprises analyzing the color image and analyzing the polarization image.

12. The method of claim 10, wherein the polarization image has a pixel resolution of at least 1200 pixels.

13. The method of claim 10, wherein the polarization image has a pixel resolution of at least 4800 pixels. The method of claim 10 , wherein the polarization image has a pixel resolution of at least 30,000 pixels.

15. The method of claim 1, wherein the mobile device has a front surface and a back surface, wherein the front surface includes a screen, and wherein the SCP camera is located on the front surface. The method of claim 15 , wherein the rear surface comprises multiple cameras.

17. The method of claim 1, wherein the SCP camera comprises a lens having an effective focal length (EFL) in the range of 2-10 mm.

18. The method of claim 17, wherein the EFL is in the range of 3-6 mm.

19. The method of claim 1, wherein the single SCP image sensor has a full image sensor diagonal (SD) in the range of 3-17 mm.

20. The method of claim 19, wherein the SD is in the range of 4-10 mm.

21. The method of claim 1, wherein the single SCP image sensor has a pixel resolution ranging from 5 megapixels (MP) to 400 MP.

22. The method of claim 21, wherein the pixel resolution is in the range of 8MP to 50MP.

23. The method of claim 21, wherein the pixel resolution is 12MP.

24. The method of claim 21, wherein the pixel resolution is 48MP.

25. The method of any one of claims 1-24, wherein the mobile device comprises a processor configured to perform said analyzing captured SCP images.

26. The method of any one of claims 1-25, wherein the mobile device includes a memory for storing a first set of facial features of the owner.

27. The method of any one of claims 1-25, wherein the mobile device comprises a communication module configured to retrieve a first set of facial features of the owner.

28. The method of claim 26 or 27, wherein analyzing the captured SCP image comprises calculating a second set of facial features and comparing the second set of facial features to the first set of facial features of the owner.

29. The method of any one of claims 1 to 27, wherein the mobile device is a smartphone.

30. The method of any one of claims 1 to 27, wherein the mobile device is a tablet computer.

31. A color polarization (CP) image sensor comprising: a first plurality, i.e., N1, of color pixel units, each color pixel unit including at least a first color pixel and a second color pixel; a second plurality of N2 polarization pixel units, each polarization pixel unit including at least a first polarization pixel and a second polarization pixel; and A single metasurface layer, wherein the single metasurface layer covers the light conversion regions of the first plurality of color pixel units and the second plurality of polarization pixel units; wherein the single metasurface layer is used to route light of a first color to the first color pixel and to route light of a second color to the second color pixel; and The single metasurface layer is used to route light of a first polarization to the first polarization pixel and to route light of a second polarization to the second polarization pixel.

32. The CP image sensor according to claim 31, wherein the single metasurface layer comprises nanorods made of at least two different materials, the at least two different materials having a lower refractive index n L and a higher refractive index n H . The CP image sensor according to claim 31 , wherein N1 ≥ 4N2 . The CP image sensor according to claim 31 , wherein N1 ≥ 16N2. The CP image sensor according to claim 31 , wherein N1 ≥ 64N2. The CP image sensor according to claim 31 , wherein N1 ≥ 128N2. The CP image sensor according to claim 31 , wherein N1 ≥ 256N2. The CP image sensor according to claim 31 , wherein N1 ≥ 1024 N2 . The CP image sensor according to claim 31 , wherein N1 ≥ 4096N2.

40. The CP image sensor of claim 31, wherein each color pixel unit includes a first color pixel, a second color pixel, and a third color pixel.

41. The CP image sensor of claim 40, wherein the first color pixel, the second color pixel, and the third color pixel are a red pixel, a green pixel, and a blue pixel.

42. The CP image sensor according to claim 31, wherein the CP image sensor is configured to detect polarization in at least two different directions.

43. The CP image sensor according to claim 42, wherein the CP image sensor is configured to detect 0-degree linear polarization, 45-degree linear polarization, 90-degree linear polarization, and 135-degree linear polarization.

44. The CP image sensor according to claim 42, wherein the CP image sensor is used to detect left-handed circular polarization and right-handed circular polarization.

45. The CP image sensor according to claim 31, wherein a height H of the single metasurface layer is in the range of 50 nm-5 μm. The CP image sensor according to claim 45 , wherein the height H is in the range of 200 nm-2 μm.

47. The CP image sensor according to claim 32, wherein n L In the range of 1-2.5, and n H In the range of 1.25-5.

48. The CP image sensor according to claim 32, wherein n L In the range of 1-1.75, and n H In the range of 2-4.

49. The CP image sensor according to claim 31, wherein the first plurality of color pixel units include a plurality of color pixels and the second plurality of polarization pixel units include a plurality of polarization pixels, the plurality of color pixels and the plurality of polarization pixels have the same pixel pitch P, and the pixel pitch P is in the range of 0.25-5 μm.

50. The CP image sensor according to claim 49, wherein the pixel pitch P is in the range of 0.35-2 μm.

51. The CP image sensor of claim 31, wherein the CP image sensor has a pixel resolution ranging from 5 megapixels (MP) to 400 MP.

52. The CP image sensor of claim 51, wherein the pixel resolution is in the range of 8MP to 50MP.

53. The CP image sensor according to claim 51, wherein the pixel resolution is 12MP.

54. The CP image sensor according to claim 51, wherein the pixel resolution is 48MP.

55. The CP image sensor of claim 31, wherein the CP image sensor is included in a camera comprising a lens having an effective focal length (EFL) in the range of 2 mm to 10 mm.

56. The CP image sensor of claim 55, wherein the EFL is in the range of 3-6 mm.

57. The CP image sensor of claim 31, wherein the CP image sensor has a full image sensor diagonal (SD) in the range of 3-17 mm.

58. The CP image sensor of claim 57, wherein the SD is in the range of 4-10 mm.

59. The CP image sensor according to any one of claims 31-58, wherein the CP image sensor is included in a mobile device.

60. The CP image sensor of claim 59, wherein the mobile device has a front surface including a screen, and the CP image sensor is included in a camera located on the front surface.

61. The CP image sensor according to claim 59, wherein the mobile device has a front surface including a screen and a rear surface opposite to the front surface, and the CP image sensor is included in a camera located on the rear surface.

62. The CP image sensor of claim 59, wherein the mobile device is a smartphone.

63. The CP image sensor of claim 59, wherein the mobile device is a tablet computer.

Images