Sensors, systems and methods for compact face identification polarization cameras

Abstract

Methods for identifying an owner of a mobile device for the purpose of unlocking the mobile device, comprising: including in the mobile device a sparse color-polarization (SCP) camera that has a single SCP image sensor configured and operational to capture SPC images, wherein the SCP image sensor includes a first plurality N1 of color pixel units and a second plurality N2 of polarization pixel units, and wherein N1 ≥ 4N2; using the SCP camera to capture a SCP image of a user; analysing the captured SCP image to determine whether the user is the owner of the mobile device; and if the user is identified as the owner, unlocking the mobile device.

Description

[0001] SENSORS, SYSTEMS AND METHODS FOR COMPACT FACE IDENTIFICATION POLARIZATION CAMERAS

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] This application is related to and claims priority from US Provisional Patent Application No. 63 / 482,847 filed February 2, 2023, which is incorporated herein by reference in its entirety.

[0004] FIELD

[0005] Embodiments disclosed herein relate in general to biometric security methods and in particular to unlocking mobile devices with biometric security methods.

[0006] BACKGROUND

[0007] Mobile handheld electronic devices (mobile devices in the following) such as smartphones, tablets, headsets, smartwatches, smart glasses etc., which are configured to use biometric methods to unlock (or to open) a mobile device are known. A popular biometric method is face unlock (or face ID). It uses image data of a face of an owner of the mobile device to extract biometric face features that are unique (or characteristic) to the owner. Based on the biometric face features, the mobile device is unlocked when the owner approaches.

[0008] FIG. 1A shows an example of a known mobile device 100 that is configured to support face unlock. Mobile device 100 has a rear surface 102 which is in general pointed away from a user. Mobile device 100 has a front surface 104 which is in general pointed towards the user. Front surface 104 includes a first selfie (or front) camera 110 having a first selfie camera field- of-view (FOVi) 112. First selfie camera 110 may be a color (or C) camera as known in the art, including for example a C image sensor such as a RGB image sensor 150 (FIG. 1C). In other examples, C camera may include a C image sensor such as a Red, Green, Blue, White (RGBW), Cyan, Magenta, Yellow (CMY), Red, Blue, White (RBW), etc. In the following we refer to first selfie camera 110 as “selfie C camera” 110. Front surface 104 may also include a second selfie camera 120 having a second selfie camera FOV2 122. As shown, FOVi 112 and FOV2 122 overlap. In some examples, second selfie camera 120 may be a depth camera as known in the art that provides depth image data. In other examples, second selfie camera 120 may be a C camera that captures C image data. First selfie camera 110 and second selfie camera 120 form a stereo camera that is operational to provide depth image data. In the following, we refer to second selfie camera 120 as “selfie depth camera” 120. Front surface 104 may include a screen 124. Rear surface 102 may include a multi-camera 126 as known in the art. The various cameras have dimensions (or “sizes”) compatible with slim and compact mobile devices.

[0009] FIG. IB shows a known method for face unlock numbered 130. In a first step 132, a user approaches a mobile device (e.g. a smartphone) like mobile device 100, configured to perform face unlock. In a second step 134, the mobile device captures a selfie image (or a video stream of selfie images) including a face such as a face 114 (FIG. 1A) of the user. In a first face unlock example, only selfie C camera 110 is used to capture a selfie C image. We refer to the first face unlock example as “single-camera face unlock”. In a second face unlock example, selfie C camera 110 and selfie depth camera 120 are used to capture a selfie C image and a selfie depth image respectively. We refer to the second face unlock example as “dual-camera face unlock”. In a third step 136, the mobile device analyses the selfie C image to identify an owner of the mobile device. For dual-camera face unlock, in a fourth step 138, the mobile device analyses the selfie depth image to identify the owner of the mobile device. In some examples of dual-camera face unlock, only the selfie depth image is used to identify the owner of the mobile device. In other examples of dual-camera face unlock, the selfie depth image is fused (or “merged”) with the selfie C image to obtain a fused “C-depth” image. The C-depth image is then analysed to identify the owner of the mobile device. For identifying an owner of the mobile device, in general a set of face features is extracted, and the set of face features is compared to known set of face features of the owner, i.e. a set of face features that were previously captured by, and stored on, the mobile device. In a fifth step 140 and in case the comparison found that the extracted set of face features is identical with the known face features of the owner, the mobile device is unlocked. In case the comparison found that the extracted set of face features differs significantly from the known face features of the owner, the mobile device is not unlocked.

[0010] An advantage of single-camera face unlock is that it can be implemented using relatively simple (and / or inexpensive) and compact camera hardware. An advantage of dualcamera face unlock over single-camera face unlock is that it provides a higher degree of security to an owner of the mobile device. A higher degree of security means here that a probability of incorrectly unlocking the mobile device is lower for dual-camera face unlock than for single-camera face unlock. Incorrectly unlocking the mobile device means for example that the mobile device is unlocked even for a person not being an owner of the mobile device, or that the mobile device is unlocked although the owner’s face is not present in the scene (referred to as “face spoofing”). For face spoofing, an image of the owner or a face mask resembling the owner may be positioned into FOVi and FOV2. The higher degree of security is achieved by using additional image information. Here, the additional image information is depth image information. A disadvantage of dual-camera face unlock is that its implementation requires relatively complex (or expensive) and large camera hardware. Specifically, it requires two cameras (instead of one camera), which is not beneficial in terms of cost, and it covers a larger area of the screen of the mobile device, which is not beneficial in terms of industrial design. In general, for face ID a level of security is measured by means of statistical parameter such as equal error rate (“EER”), true positive rate (“TPR”) or False Positive Rate (“FPR”) as known in the art, or derivatives thereof such as receiver operating characteristic (“ROC”) curve and Area under the ROC Curve (“AUC”). Hereby, a relatively low value for EER (i.e. relatively close to zero) and a relatively high value for AUC (i.e. relatively close to one) correspond to a relatively high level of security. In general, the "Fast IDentity Online" (or “FIDO”) Alliance defines the security standards used for mobile devices.

[0011] FIG. 1C shows a known RGB image sensor 150 in a top view. Included in a RGB camera, the optical axis of the RGB camera is oriented perpendicular to RGB image sensor 150. RGB image sensor 150 includes a plurality of single pixels (red-R, green-G and blue-B) like single R pixel 152. In general, in a top view a single pixel has a square shape. RGB image sensor 150 comprises a plurality of pixel units like a pixel unit 154. Exemplarily, pixel unit 154 includes one R pixel, one B pixel and two G pixels. Pixel unit 154 represents the smallest pixel unit (or “assembly”) that can be used as a building block to construct (“assemble”) RGB image sensor 150. As shown, RGB image sensor 150 can be formed by repeating pixel units such as pixel unit 154, as shown for pixel units 156, 158, 160 and 162.

[0012] FIG. ID shows a known polarization image sensor 170 in a top view. Included in a polarization camera, an optical axis of the polarization camera is oriented perpendicular to polarization image sensor 170. Polarization image sensor 170 includes a plurality of single pixels like single pixel 172. Polarization image sensor 170 comprises a plurality of pixel units like a pixel unit 174. Pixel unit 174 includes one pixel measuring incoming light having a 90 degree linear polarization (marked “90°”), one pixel measuring incoming light having a 45 degree linear polarization (marked “45°”), one pixel measuring incoming light having a 135 degree linear polarization (marked “135°”) and one pixel measuring incoming light having a 0 linear degree polarization (marked “0°”). In general, a polarization filter is applied on top of each pixel. Pixel unit 174 represents the smallest pixel unit that can be used as a building block to construct polarization image sensor 170. As shown, polarization image sensor 170 can be formed by repeating pixel units such as pixel unit 174, as shown for pixel units 175, 176, 177 and 178.

[0013] It is known that a polarization image sensor such as polarization image sensor 170 included in a polarization camera can provide additional image information. Here, the additional image information is polarization image information. The polarization image information can be used in a dual-camera face unlock (instead of the depth image information) to provide a higher degree of security.

[0014] Face features represent characteristic shapes, distances between shapes, etc. that are present in color images and polarization images respectively. For example, contrast in the color image that allows for identification of a particular face feature may be based on a color information, whereas contrast in the polarization image that allows for identification of a particular face feature may be based on a degree of linear polarization (“DOLP”) or on an angle of linear polarization (“AOLP”). Contrast in the polarization image may also be based on a degree of circular polarization (“DOCP”) or on an angle of circular polarization (“AOCP”). Image sensors that are operational to measure DOCP and AOCP are for example described in US patent 11841522B2.

[0015] For example, it is known that polarization image data can be used to determine surface normals of a surface of a captured object. This fact can be used to prevent particular examples of face spoofing. Consider a face spoofing attack using a photo of a face of an owner printed on a piece of paper. In a C image, it is sometimes very hard (both for humans and algorithms) to distinguish between an image of an actual face of the owner and an image of a photo of the face of the owner. However, based for example by using information on a surface normal such as extractable from a polarization image, one can effectively differentiate between an image of the actual face of the owner (exhibiting a relatively large variation in terms of surface normal) and an image of a photo of the face of the owner (exhibiting a relatively small variation in terms of surface normal). It is known that screens (or “monitors”) often emit polarized light. Thus, polarization image data can be used to distinguish between a real face and a photo of a face as shown on a screen.

[0016] There is need for and it would be beneficial to have a single-camera face unlock that can provide a higher degree of security than provided with known single-camera face unlock.

[0017] SUMMARY In various exemplary embodiments, there is provided a method, comprising: providing a mobile device comprising a sparse color-polarization (SCP) camera that includes a single SCP image sensor configured and operational to capture SPC images, wherein the SCP image sensor includes a first plurality Ni of color pixel units and a second plurality N2 of polarization pixel units, and wherein Ni > 4N2; capturing a SCP image of a user; analysing the captured SCP image to determine whether the user is an owner of the mobile device; and if the user is identified as the owner, unlocking the mobile device.

[0018] In some examples, Ni > 4N2. In some examples, Ni > I6N2. In some examples, Ni > 64N2. In some examples, Ni > 128N2. In some examples, Ni > 256N2. In some examples, Ni > 1024N2. In some examples, Ni > 4096N2.

[0019] In some examples, the polarization pixel units are operational to capture linear polarization information. In some examples, the polarization pixel units are operational to capture circular polarization information.

[0020] In some examples, the SCP images are used to create a color image and a polarization image.

[0021] In some examples, the analysing of the captured SCP image includes analysing the color image and analysing the polarization image.

[0022] In some examples, the polarization image has a pixel resolution of at least 1200 pixels. In some examples, the polarization image has a pixel resolution of at least 4800 pixels. In some examples, the polarization image has a pixel resolution of at least 30000 pixels.

[0023] In some examples, the mobile device has a front surface and a rear surface, the front surface includes a screen, and the SCP camera is located at the front surface. In some such examples, the rear surface includes a multi-camera.

[0024] In some examples, the SCP camera includes a lens having an effective focal length (EFL) in the range of 2-10mm. In some examples, the EFL is in the range of 3-6mm.

[0025] In some examples, the single SCP image sensor has a full image sensor diagonal (SD) in the range of 3- 17mm. In some examples, the SD is in the range of or 4- 10mm.

[0026] In some examples, the single SCP image sensor has a pixel resolution in the range of 5 megapixel (MP) to 400MP. In some examples, the pixel resolution in the range of 8MP to 50MP. In some examples, the pixel resolution is 12MP. In some examples, the pixel resolution is 48MP.

[0027] In some examples, the mobile device includes a processor configured to perform the analysing of the captured SCP image. In some examples, the mobile device includes a memory for storing a first set of face features of the owner. In some examples, the mobile device includes a communication module configured to for retrieve a first set of face features of the owner.

[0028] In some examples, the analysing of the captured SCP image includes calculating a second set of face features and comparing the second set of face features with the first set of face features of the owner.

[0029] In various exemplary embodiments, there is provided a color-polarization (CP) image sensor, comprising: a first plurality Ni of color pixel units, each color pixel unit including at least a first color pixel and a second color pixel; a second plurality N2 of polarization pixel units, each polarization pixel unit including at least a first polarization pixel and a second polarization pixel; and a single metasurface layer, wherein the single metasurface layer covers a photo-conversion area of both the color pixel units and of the polarization pixel units, wherein the single metasurface layer is operational to route light of a first color into a first color pixel, and light of a second color into a second color pixel, and wherein the single metasurface layer is operational to route light of a first polarization into a first polarization pixel, and light of a second polarization into a second polarization pixel.

[0030] In some examples, the single metasurface layer includes nano-pillars made of at least two different materials having two different refractive indices.

[0031] In some examples, Ni > 4N2. In some examples, Ni > I6N2. In some examples, Ni > 64N2. In some examples, Ni > 128N2. In some examples, Ni > 256N2. In some examples, Ni > 1024N2. In some examples, Ni > 4096N2.

[0032] In some examples, the CP image sensor is operational to detect three different colors. In some examples, the three detected colors are red, green and blue.

[0033] In some examples, the CP image sensor is operational to detect two different polarization directions. In some examples, the CP image sensor is operational to detect four different polarization directions.

[0034] In some examples, the CP image sensor is operational to detect 0-degree linear polarization, 45-degree linear polarization, 90-degree linear polarization and 135-degree linear polarization.

[0035] In some examples, the CP image sensor is operational to detect left-handed circular polarization and right-handed circular polarization.

[0036] In some examples, the single metasurface layer has a height H in the range of 50nm - 5pm. In some examples, H is in the range of 200nm - 2um. In some examples, a lower refractive index n is in the range of 1 to 2.5, and a higher refractive index nu is in the range of 1.25 to 5. In some examples, n is in the range of 1 to 1.75, and nn is in the range of 2 to 4.

[0037] In some examples, the color pixel units include color pixels and the polarization pixel units include polarization pixels, the color pixels and the polarization pixels having an identical pixel pitch P in the range of 0.25pm - 5pm. In some examples, P is in the range of 0.35pm - 2pm.

[0038] In some examples, a pixel resolution of the SCP image sensor is in the range of 5MP to 400MP. In some examples, the pixel resolution is in the range of 8MP to 50MP. In some examples, the pixel resolution is 12MP. In some examples, the pixel resolution is 48MP.

[0039] In some examples, the CP image sensor is included in a camera that has a lens with an EFL in the range of 2- 10mm. In some examples, EFL is in the range of 3-6mm.

[0040] In some examples, the CP image sensor has a full image SD in the range of 3-15mm. In some examples, SD is in the range of or 4- 10mm.

[0041] In some examples, the CP image sensor is included in a mobile device. In some examples, the mobile device has a front surface including a screen, and the CP image sensor is included in a camera located at the front surface.

[0042] In some examples, the mobile device is a smartphone. In some examples, the mobile device is a tablet.

[0043] BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein and should not be considered limiting in any way. Like elements in different drawings may be indicated by like numerals. Elements in the drawings are not necessarily drawn to scale. In the drawings:

[0045] FIG. 1A shows schematically a mobile device configured to perform face unlock as known in the art;

[0046] FIG. IB shows a method to perform face unlock as known in the art;

[0047] FIG. 1C shows a RGB image sensor as known in the art;

[0048] FIG. ID shows a polarization image sensor as known in the art;

[0049] FIG. IE shows a sparse color-polarization (SCP) image sensor as known in the art;

[0050] FIG. 2A shows an example of a SCP selfie image for performing single-camera face unlock as disclosed herein;

[0051] FIG. 2B shows an example of a SCP image sensor as disclosed herein in a top view;

[0052] FIG. 2C shows the SCP image sensor of FIG. 2B in a cross-sectional side view;

[0053] FIG. 2D shows a metasurface layer included in the SCP image sensor of FIG. 2B in a cross-sectional side view;

[0054] FIG. 2E shows parts of the metasurface layer of FIG. 2D in a top view;

[0055] FIG. 2F shows other parts of the metasurface layer of FIG. 2D in a top view;

[0056] FIG. 3 shows a method for single-camera face unlock as disclosed herein;

[0057] FIG. 4 shows schematically an embodiment of a mobile device configured to perform single-camera face unlock disclosed herein.

[0058] DETAILED DESCRIPTION

[0059] FIG. IE shows a known sparse color-polarization (SCP) image sensor 180. SCP image sensor 180 includes single pixels like a single pixel 182 arranged in a first plurality of color pixel units 184 (similar to unit 154) and a second plurality of polarization pixel units 186 (similar to unit 174). For example and as shown, color pixel units 184 are RGB pixel units. Polarization pixel units 186 are “sparsely” integrated between color pixel units 184. “Sparsely integrated” refers here to the fact that SCP image sensor 180 includes more color pixel units 184 than polarization pixel units 186. As shown and for example, color pixel units 184 are a position (or location) 188 alongside polarization pixel units 186 at a position 190. Overall, SCP image sensor 180 may include N individual pixels, “individual” referring to each R, G, B or 0°, 45°, 90°, or 135° pixel. Of the total N individual pixels in the SCP image sensor, the proportion “r” of polarized pixels may be r < 1 / 21, i = 2, 3, 4 ... etc. Accordingly, SCP image sensor 180 includes N x r single polarization pixels and N x (1- r) single color pixels. In the example of FIG. IE, r = 1 / 4. In other examples, r may be r < 1 / 16 or r < 1 / 64 etc. In the following, we refer to r as the “sparsity” of the polarization pixels.

[0060] In yet other examples (not shown), a polarization pixel unit like unit 186 may be also be configured to capture color image data. For example, such polarization pixel units may also be covered with color filters, in addition to polarization filters. In such examples, a single pixel (not shown) can measure both a polarization signal and a color signal. That is, a single pixel can measure one polarization direction and one color.

[0061] FIG. 2A illustrates schematically an example of a captured SCP selfie image 200 for performing single-camera face unlock as disclosed herein. SCP selfie image 200 is captured by a sparse CP selfie camera including a SCP image sensor such as SCP image sensor 180. The SCP selfie camera may be a front camera in a mobile device. Captured SCP selfie image 200 may include a face of the user taking the selfie. SCP selfie image 200 includes color (e.g. RGB) image data 202 captured by color pixel units such as units 184 and polarization image data 204 captured by polarization pixel units such as units 186. Exemplarily, here polarization image data represented by rectangles 204 is available at only nine locations in SCP selfie image 200. In general, the number of such locations will be much higher, e.g. at least hundreds, or thousands, or even hundreds of thousands of locations in a SCP image.

[0062] It is noted that for each image pixel (or image point) within SCP selfie image 200, only color image data or only polarization image data is available. It is noted that in dark (or “low- light”) scenes, polarization image data often provides a stronger contrast than color image data. For example, DOLP image data or AOLD image data show stronger contrast than color image data. It is known that single-camera face unlock performs significantly worse in dark scenes when compared to dual-camera face unlock. This is because for dual-camera face unlock, the scene is often artificially illuminated, e.g. in the near-infrared (NIR) spectrum, which is not visible for humans. In general, artificial illumination of the scene is impossible or undesired for single-camera face unlock. Therefore, the stronger contrast of polarization image data is beneficial for performing single-camera face unlock.

[0063] Moving now to the processing of the captured image data, after performing RGB demosaicing on SCP selfie image 200 as known in the art, three values (one for each R, G, B) are available for each single pixel included in a region of RGB image data 202. After performing polarization demosaicing on SCP selfie image 200 as known in the art, four values (one for each 90°, 45°, 135° and 0°) are available for each single pixel included in a region of polarization image data 204. In some examples, a pixel unit like polarization pixel unit 186 may be considered to be one, single “larger” pixel. For this pixel unit including one larger pixel, four values (one for each 90°, 45°, 135° and 0°) are available after performing polarization demosaicing. It is noted that two separate images can be created from SCP selfie image 200, for example, a polarization image that includes only polarization image data, and a color image that includes only color image data.

[0064] When using the SCP selfie camera for photography, i.e. for capturing output images that are output (or “displayed”) to a user, in general only color pixel units are used, while polarization pixel units are not used. For photography, a relatively high sparsity is beneficial for achieving a relatively high image quality (“IQ”), as more color pixel units contribute to the output image. For performing single-camera face unlock as disclosed herein (FIG. 3), a relatively high sparsity of e.g. N > 64, or N > 256, or N > 1024 or even N > 4096 may be sufficient to achieve a relatively high level of security.

[0065] A CP image sensor such as SCP image sensor 180 or SCP selfie image sensor 220 or CP image sensor 225 (see below) may have a pixel (or “spatial”) resolution in the range of 2.5 megapixel (MP) to 400MP, typically in the range of 8MP to 50MP, for example 12MP or 48MP. A width : height ratio of the SCP image sensor may be 4:3 or 16:9. A polarization image that includes only polarization image data may have a pixel resolution in the range of about 500 pixel to 15MP. For example, a pixel resolution may be about 40x30 (1200 pixel) or about 80x60 (4800 pixel) or about 200x150 (30000 pixel) etc.

[0066] In terms of camera and image sensor parameters, a camera including a CP or a SCP image sensor may include a lens having an EFL in the ranges of 2mm - 15mm or 3mm - 6mm, and the CP or the SCP image sensor may have a (full) image sensor diagonal SD in the ranges of 2mm - 17mm or 4mm - 10mm.

[0067] FIG. 2B shows an example of a SCP selfie image sensor disclosed herein and numbered 220 in a top view. SCP selfie image sensor 220 includes color image sensor regions 222 which include color pixel units as disclosed herein, and polarization image sensor regions 224 which include polarization pixel units as disclosed herein.

[0068] FIG. 2C shows exemplary a color-polarization image sensor 225 as disclosed herein in a cross-sectional side view. The cross section shows a first color pixel unit 226, a polarization pixel unit 228 and a second color pixel unit 230. Along the z-axis in the z-x coordinate system shown, the pixel units include a metasurface layer 232, a (optional) spacer layer 234, a (optional) filter layer 236, and a pixel layer 238. Color image sensor regions 222 and polarization image sensor regions 224 are both covered by metasurface layer 232.

[0069] Along the x-axis, metasurface layer 232 includes a first color-router 240 in first color pixel unit 226, a polarization-router 242 in polarization pixel unit 228 and a second color-router 244 in second color pixel unit 230. First color-router 240 and second color-router 244 may be “metasurface color-routers” (or “metasurface nano-prisms”) as known in the art. Metasurface color-routers are operational to route color to a particular pixel region or into a particular pixel. Metasurface color-routers are described for example in “Optical design of dispersive metasurface nano-prism structure for high sensitivity CMOS image” by Chulsoo Choi et al., IEEE International Electron Devices Meeting (IEDM), December 9- 13,.2022023. Metasurface color-routers may be used instead of, or in conjunction with, color filters, which is beneficial, since a higher amount of light signal can be harvested by a CP selfie image sensor 225 with metasurface color-filters when compared to color image sensor regions using (only) color filters.

[0070] Metasurface lenses (or “metalenses”) that route particular polarization directions into a particular image sensor region are known, for example as used in the product “Polar ID” by company Metalenz. Polarization-router 242 may include “metasurface polarization-routers” (or simply “polarization-routers”), i.e. polarization- sensitive metalenses, that route particular polarization directions into a particular pixel region or into a particular pixel. Polarizationrouters may be used instead of, or in conjunction with, polarization filters, which is beneficial as a higher amount of light signal can be harvested by a SCP selfie image sensor 220 with polarization-router when compared to polarization image sensor regions using (only) polarization filters. Both color-routers and polarization-routers may be referred to as “nanorouters”.

[0071] In some examples, CP image sensor 225 may be a sparse CP image sensor, e.g. like SCP image sensor 220. In such examples, CP image sensor 225 may include a first plurality Ni of color pixel units and a second plurality N2 of polarization pixel units, and wherein Ni > N2.

[0072] In other examples, CP image sensor 225 may not be a sparse CP image sensor. In such examples, CP image sensor 225 may include a first plurality N 1 of color pixel units and a second plurality N2 of polarization pixel units, and wherein Ni = N2.

[0073] In some examples, CP image sensor 225 may be included in a camera located at a front surface such as front surface 104. In other examples, CP image sensor 225 may be included in a camera located at a rear surface of a mobile device such as rear surface 102.In general, a CP image sensor such as SCP image sensor 180 or SCP image sensor 220 or CP image sensor 225 may be operational to harvest visible light. In some examples, a SCP image sensor may be operational to harvest light in the near infrared (“NIR”) range, i.e. light in a wavelength range of about 750nm - lOOOnm. In other cases, a CP image sensor may be operational to harvest light in the short-wave infrared (“SWIR”) range, i.e. light in a wavelength range of about lOOOnm - 1800nm. In yet other examples, a CP image sensor may be operational to harvest visible light, IR light and SWIR light, i.e. light in a wavelength range of about 450nm - 1800nm. In all such examples, “color” may refer to a particular wavelength band within the visible and NIR and SWIR wavelength region respectively.

[0074] Optional spacer 234 may be used to allow sufficient spatial splitting of, respectively, the spectral and polarization components of the light. In some examples, optional filter 236 may be a known color filter that compensates for a non-ideal spatial splitting of the light’s spectral components. In other examples, optional filter 236 may be a known polarization filter that compensates for a non-ideal spatial splitting of the light’ s polarization components. In yet other examples, optional filter 236 may be a color filter in color pixel units, and it may be a polarization filter in polarization pixel units. Photon-electron conversion (or “photon harvesting”) takes place in pixel layer 238, as known.

[0075] FIG. 2C shows schematically rays of green light (G) and of red light (R) impinging on CP image sensor 225 from a direction parallel to the z-axis. Exemplarily, first color pixel unit 226 and second color pixel unit 230 include a G pixel 246 and a G pixel 254 as well as a R pixel 248 and a R pixel 256 respectively. Exemplarily, polarization pixel unit 228 includes a 90-degree linear polarization (“90°”) pixel 250 as well as a 45-degree linear polarization (“45°”) pixel 252. As of first color-router 240 and second color-router 244 respectively, any G light (solid line) impinging on first color pixel unit 226 and second color pixel unit 230 is routed (or “directed”) respectively into G pixel 246 and G pixel 254, and any R light (dotted line) impinging on first color pixel unit 226 and second color pixel unit 230 is routed respectively into R pixel 248 and R pixel 256. As of polarization-router 242, any 90-degree linearly polarized light (solid line) impinging on polarization pixel unit 228 is routed into 90° pixel 250, and any 45-degree linearly polarized light (dotted line) impinging on polarization pixel unit 228 is routed into 45° pixel 252. A pixel pitch (or “pixel width”) of a single pixel is indicated by “P” and may be in the range of 0.25pm - 5pm, preferably in the range of 0.35pm - 2pm, e.g. 0.5pm, or 0.6pm or 0.7pm etc.

[0076] FIG. 2D shows exemplarily a metasurface layer 232 disclosed herein and operational to be used in a CP image sensor like CP image sensor 225 in a cross-sectional side view. Here and exemplarily, metasurface layer 232 has a symmetry axis which is oriented parallel to the x-axis. In general, metasurface layer 232 is created (or manufactured) by forming nanostructures (also referred to as “nano-pillars” or “nano-rods”) of two or more different materials. For example, a first type of nanostructure 258 may be made of a first material having a relatively low refractive index (“HL”) and a second type of nanostructure 260 may be made of a second material having a relatively high refractive index (“UH”). In general, UH > n . n may be in the range of 1 to 2.5, preferably in the range of 1 - 1.75, and numay be in the range of 1.25 to 5, preferably in the range of 2 - 4 or more. In some examples, numay be air. In FIG. 2D, each white region of metasurface layer 232 represents the first type of nanostructure 258, and each black region of metasurface layer 232 represents the second type of nanostructure 260. Two regions, each including both types of nanostructures 258 and 260 are indicated as an example. A height (“H”) of metasurface layer 232 (and of the nanostructure) may be in the range of 50nm - 5p m, preferably in the range of 200nm - 2pm. A width (“W”) of an individual nanostructure may be in the range of 20nm - 1pm, preferably in the range of 50nm - 500nm.

[0077] In some examples and as shown, both color-routers 240 and 244 as well as polarizationrouter 242 are made (or “manufactured”) from the same two or more different materials. In other words, there are material combinations that allow both the manufacturing of color-routers and of polarization-routers, so that color-routers and polarization-routers can be realized in single metasurface layer. The functionality of color-routing and polarization-routing is primarily obtained by a geometry of the nanostructure (FIG. 2E), as well as by an interaction of the nano-pillars with each other, and not necessarily by the material combination. This means that a “single” or “continuous” metasurface layer such as metasurface layer 232 can provide color-routing as well as polarization-routing capabilities. In terms of manufacturing time, process complexity and manufacturing cost, this is beneficial for manufacturing a CP image sensor 225 disclosed herein, because it allows the manufacturing of color-routers 240 and 244 as well as of polarization-router 242 in one or more identical process steps required to manufacture the single metasurface layer. That is, identical process steps can be used to manufacture both color-routers 240 and 244 as well as polarization-router 242. Here, one layer of nanostructures is used in metasurface layer 232. In other examples, a metasurface layer such as metasurface layer 232 may include two or more layers of nanostructures, i.e. a stack of nanostructures. Also to these other examples, we refer as “single” or “continuous” metasurface layer, if the functionality of color-routing and polarization-routing is introduced in identical process steps, and in particular while using the same two or more different materials.

[0078] FIG. 2E shows exemplarily color-router 240 in a top view. Exemplarily, color-router 240 may spatially split light into three different colors, R, G and B.

[0079] FIG. 2F shows exemplarily polarization-router 242 in a top view. Exemplarily, polarization-router 242 may spatially split light into four different linear polarizations, into 90°, 45°, 0° and 135°.

[0080] As shown, the geometry (or “shape”) of nanostructures included in color-router 240 and in polarization-router 242 respectively show structural differences. Specifically, nanostructures included in color-router 240 may predominantly include geometries that are rotational- symmetric along the z-axis. Nanostructures included in polarization-router 242 may predominantly include geometries which are not rotational-symmetric along the z-axis, but have a preferred directionality in the x-y plane. “Predominantly” means here that in a colorrouter there may be a larger share of rotational-symmetric geometries than in a polarizationrouter, and vice versa, that in a polarization-router there may be a larger share of preferred directionality geometries than in a color-router. Still, preferred directionality geometries may be included in the color-router, and rotational-symmetric geometries may be included in the polarization-router.

[0081] FIG. 3 shows a method for single-camera face unlock disclosed herein and numbered 300. In step 302, a user approaches a mobile device configured to perform single-camera face unlock. In step 304, the mobile device captures a SCP selfie image 200 (or a stream of selfie SCP selfie images) including a face such as face 114 of the user. In step 306, the mobile device analyses a color image created from SCP selfie image 200 to identify the face of an owner of the mobile device. In step 308, the mobile device analyses a polarization image created from SCP selfie image 200 to identify the face of an owner of the mobile device. In each of steps 306 and 308, a set of face features may be extracted from SCP selfie image 200, and the set of face features may be compared to the face features of the owner. In general, the face features extracted in step 306 and 308 are different, which is beneficial for achieving a relatively high level of security. In other examples, in step 308 a detection of face spoofing may be performed. Performing the detection of face spoofing may include analysing a polarization image to distinguish between a real face and a photo of a face as shown on a screen or a photo of a face printed on a piece of paper.

[0082] In step 310 and in case the mobile device found that a set of face features extracted in steps 306 and 308 is identical with face features of the owner of the mobile device, the mobile device is unlocked. In case the mobile device found that a set of face features extracted in steps 306 and 308 differs from face features of the owner of mobile device, the mobile device is not unlocked.

[0083] In some examples of single-camera face unlock and by performing step 306 and step 308 in one single step, the mobile device analyses SCP selfie image 200 as is. In other words, color image data and polarization image data may be analyzed in one single step.

[0084] In some examples of single-camera face unlock and by performing step 306 and step 308 in one single step, the mobile device creates a fused CP image in a first sub-step. In some of these examples, color image data is used to estimate (or artificially generate) polarization image data which is actually not present in SCP selfie image 200. In other examples, polarization image data is used to estimate color image data that is not actually present in SCP selfie image 200, after which a fused CP image is created. In an example, seven values (one for each R, G, B, 90°, 45°, 135° and 0°) may be available for each pixel of the fused CP image. One may refer to the latter as a “dense CP image”. In a second sub-step, the mobile device analyses the created image to identify a face of the owner of the mobile device. In some examples of single-camera face unlock and as a first sub-step of step 306, image processing methods known in the art such as “super-resolution” or “upsampling” are used to create a “high-resolution color image”, i.e. a color image with elevated pixel resolution. For example, the created color image may have a pixel resolution identical to the SCP selfie image pixel resolution. In a second sub-step of step 306, the mobile device analyses the high- resolution color image to identify a face of an owner of the mobile device.

[0085] In some examples of single-camera face unlock and as a first sub- step of step 308, image processing methods are used to create a “high-resolution polarization image”, i.e. a polarization image with elevated pixel resolution. For example, the created polarization image may have a pixel resolution identical to the SCP selfie image pixel resolution. In a second substep of step 308, the mobile device analyses the high-resolution polarization image to identify a face of an owner of the mobile device.

[0086] An advantage of method 300 over known single-camera face unlock methods is that it provides a higher degree of security. The higher degree of security is achieved by using additional image information. Specifically, here the additional image information is polarization image information.

[0087] An advantage of method 300 over the known dual-camera face unlock is that it can be implemented using relatively simple (or inexpensive) and compact camera hardware. Specifically, only one camera is required (instead of two cameras), which is beneficial in terms of cost, and it covers a smaller area of a screen such as screen 124, which is beneficial in terms of industrial design.

[0088] For performing method 300, a mobile device such as mobile device 400 is provided. FIG. 4 shows schematically an embodiment of a mobile device (for example, a smartphone) numbered 400 and configured to perform single-camera face unlock disclosed herein. Mobile device 400 comprises a selfie camera 410 having a selfie camera FOVs and including an image sensor 412. Image sensor 412 is a sparse CP image sensor such as SCP image sensor 180 or SCP image sensor 220 or CP image sensor 225 as defined above. Optionally, mobile device 400 further comprise a second camera, e.g. a multi-camera as known in the art. In other examples, a sparse CP image sensor may be integrated in the multi-camera.

[0089] Mobile device 400 further includes an application processor (AP) 420. AP 420 includes a face detector 422, e.g. configured to perform face detection, a color face identifier 424, e.g. configured to identify a face of an owner of mobile device 400 in a color image, a polarization face identifier 426, e.g. configured to identify a face of an owner of mobile device 400 in a polarization image and / or to detect face spoofing, and an image sensor controller 428. Mobile device 400 includes in addition a screen 430 and a memory 440. Memory 440 may e.g. be used to store a set of face features of an owner of mobile device 400. This stored set of face features may be used in step 306 and / or step 308 for single-camera face unlock. In other examples, mobile device 400 may in addition include a communication module, operational to retrieve a set of face features of an owner of mobile device 400. For example, the communication module may retrieve the set of face features of an owner of mobile device 400 from the internet, from a server, from a cloud etc.

[0090] While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.

[0091] All references mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual reference was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present application.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: providing a mobile device comprising a sparse color-polarization (SCP) camera that includes a single SCP image sensor configured and operational to capture SPC images, wherein the SCP image sensor includes a first plurality Ni of color pixel units and a second plurality N2 of polarization pixel units, and wherein Ni > 4N2; capturing a SCP image of a user; analysing the captured SCP image to determine whether the user is an owner of the mobile device; and if the user is identified as the owner, unlocking the mobile device.

2. The method of claim 1, wherein Ni >16N2.

3. The method of claim 1, wherein Ni > 64N2.

4. The method of claim 1, wherein Ni > 128N2.

5. The method of claim 1, wherein Ni > 256N2.

6. The method of claim 1, wherein Ni > 1024N2.

7. The method of claim 1, wherein Ni > 4096N2.

8. The method of claim 1, wherein the polarization pixel units are operational to capture linear polarization information.

9. The method of claim 1, wherein the polarization pixel units are operational to capture circular polarization information.

10. The method of claim 1, wherein the SCP images are used to create a color image and a polarization image.

11. The method of claim 10, wherein the analysing the captured SCP image includes analysing the color image and analysing the polarization image.

12. The method of claim 10, wherein the polarization image has a pixel resolution of at least 1200 pixels.

13. The method of claim 10, wherein the polarization image has a pixel resolution of at least 4800 pixels.

14. The method of claim 10, wherein the polarization image has a pixel resolution of at least 30000 pixels.

15. The method of claim 1, wherein the mobile device has a front surface and a rear surface, wherein the front surface includes a screen, and wherein the SCP camera is located at the front surface.

16. The method of claim 15, wherein the rear surface includes a multi-camera.

17. The method of claim 1, wherein the SCP camera includes a lens having an effective focal length (EFL) in the range of 2- 10mm.

18. The method of claim 17, wherein the EFL is in the range of 3-6mm.

19. The method of claim 1, wherein the single SCP image sensor has a full image sensor diagonal (SD) in the range of 3- 17mm.

20. The method of claim 19, wherein the SD is in the range of or 4- 10mm.

21. The method of claim 1, wherein the single SCP image sensor has a pixel resolution in the range of 5 megapixel (MP) to 400MP.

22. The method of claim 21, wherein a pixel resolution in the range of 8MP to 50MP.

23. The method of claim 21, wherein a pixel resolution is 12MP.

24. The method of claim 21, wherein a pixel resolution is 48MP.

25. The method of any of the claims 1-24, wherein the mobile device includes a processor configured to perform the analysing of the captured SCP image.

26. The method of any of the claims 1-25, wherein the mobile device includes a memory for storing a first set of face features of the owner.

27. The method of any of the claims 1-25, wherein the mobile device includes a communication module configured to for retrieve a first set of face features of the owner.

28. The method of claim 26 or 27, wherein the analysing of the captured SCP image includes calculating a second set of face features and comparing the second set of face features with the first set of face features of the owner.

29. The method of any of the claims 1-27, wherein the mobile device is a smartphone.

30. The method of any of the claims 1-27, wherein the mobile device is a tablet.

31. A color-polarization (CP) image sensor, comprising: a first plurality Ni of color pixel units, each color pixel unit including at least a first color pixel and a second color pixel; a second plurality N2 of polarization pixel units, each polarization pixel unit including at least a first polarization pixel and a second polarization pixel; and a single metasurface layer, wherein the single metasurface layer covers a photo-conversion area of both the color pixel units and of the polarization pixel units, wherein the single metasurface layer is operational to route light of a first color into a first color pixel, and light of a second color into a second color pixel, and wherein the single metasurface layer is operational to route light of a first polarization into a first polarization pixel, and light of a second polarization into a second polarization pixel.

32. The CP image sensor of claim 31, wherein the single metasurface layer includes nanopillars made of at least two different materials having respectively a lower refractive index n and a higher refractive index UH.

33. The CP image sensor of claim 31, wherein Ni > 4N2.

34. The CP image sensor of claim 31, wherein Ni > I6N2.

35. The CP image sensor of claim 31, wherein Ni > 64N2.

36. The CP image sensor of claim 31, wherein Ni > 128N2.

37. The CP image sensor of claim 31, wherein Ni > 256N2.

38. The CP image sensor of claim 31, wherein Ni > 1024N2.

39. The CP image sensor of claim 31, wherein Ni > 4096N2.

40. The CP image sensor of claim 31, wherein each color pixel unit includes a first color pixel, a second color pixel and a third color pixel.

41. The CP image sensor of claim 40, wherein the three color pixel are red pixel, green pixel and blue pixel.

42. The CP image sensor of claim 31, wherein the CP image sensor is operational to detect at least two different polarization directions.

43. The CP image sensor of claim 42, wherein the CP image sensor is operational to detect 0-degree linear polarization, 45-degree linear polarization, 90-degree linear polarization and 135-degree linear polarization.

44. The CP image sensor of claim 42, wherein the CP image sensor is operational to detect left-handed circular polarization and right-handed circular polarization.

45. The CP image sensor of claim 31, wherein the single metasurface layer has a height H in the range of 50nm - 5|im.

46. The CP image sensor of claim 45, wherein H is in the range of 200nm - 2|im.

47. The CP image sensor of claim 32, wherein n is in the range of 1- 2.5, and wherein nu is in the range of 1.25 - 5.

48. The CP image sensor of claim 32, wherein n is in the range of 1- 1.75, and wherein a nu is in the range of 2 - 4.

49. The CP image sensor of claim 31 , wherein the color pixel units include color pixels and the polarization pixel units include polarization pixels, the color pixels and the polarization pixels having an identical pixel pitch P in the range of 0.25-5pm.

50. The CP image sensor of claim 49, wherein P is in the range of 0.35-2pm.

51. The CP image sensor of claim 31, wherein the CP image sensor has a pixel resolution in the range of 5 megapixel (MP) to 400MP.

52. The CP image sensor of claim 51, wherein the pixel resolution is in the range of 8MP to 50MP.

53. The CP image sensor of claim 51, wherein the pixel resolution is 12MP.

54. The CP image sensor of claim 51, wherein the pixel resolution is 48MP.

55. The CP image sensor of claim 31, wherein the CP image sensor is included in a camera, wherein the camera includes a lens having an effective focal length (EFL) in the range of 2mm to 10mm.

56. The CP image sensor of claim 55, wherein EFL is in the range of 3 -6mm.

57. The CP image sensor of claim 31, wherein the CP image sensor has a full image sensor diagonal (SD) in the range of 3- 17mm.

58. The CP image sensor of claim 57, wherein SD is in the range of or 4-10mm.

59. The CP image sensor of any of the claims 31-58, wherein the CP image sensor is included in a mobile device.

60. The CP image sensor of claim 59, wherein the mobile device has a front surface including a screen, and wherein the CP image sensor is included in a camera located at the front surface.

61. The CP image sensor of claim 59, wherein the mobile device has a rear surface opposing a front surface including a screen, and wherein the CP image sensor is included in a camera located at the rear surface.

62. The CP image sensor of claim 59, wherein the mobile device is a smartphone.

63. The CP image sensor of claim 59, wherein the mobile device is a tablet.

Images