Imaging system

Abstract

An imaging system (111) is proposed, comprising - at least one pattern illumination source (112) configured for emitting at least one light pattern comprising a plurality of light beams having wavelengths in a short-wavelength infrared spectral region, wherein the wavelengths of the light are from 1000 nm to 1200 nm or from 1300 nm to 1500 nm; and - at least one image sensor (118) configured for generating at least one pattern image while the pattern illumination source (112) emits the light pattern, wherein the image sensor (118) is at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region.

Description

Technical Field

[0001] This invention relates to an imaging system, an apparatus for authenticating a user of a device, a method for authenticating a user, a computer program, a computer-readable storage medium, a non-transient computer-readable medium, and several applications. The apparatus, method, and applications according to the invention can be specifically used, for example, in various fields such as daily life, security technology, gaming, transportation technology, production technology, photography (e.g., digital or video photography for artistic, documentation, or technical purposes), safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, and medical technology, or in science. However, other applications are also possible. Background Technology

[0002] Available authentication systems in mobile devices (e.g., smartphones, tablets, etc.) include a receiver, such as at least one camera. The mobile device typically has a front-facing display, such as an organic light-emitting diode (OLED) area and / or a quantum dot light-emitting diode (QLED) area. The receiver can be positioned behind the front-facing display. Furthermore, light emitters, such as projectors, or one or more light-emitting diodes and / or lasers, are used in such authentication devices, and these light emitters can be positioned behind the display. Typically, the light emitter projects a pattern, such as a dot pattern, onto a target (e.g., a face), and the receiver (e.g., a camera) captures the image projected onto the user, and a processor determines material data. If the material is classified as skin, it is identified as human; otherwise, it is identified as a spoofed target. The pattern generated by the light projector can be designed for 3D algorithms; that is, the pattern can be designed to easily solve the so-called correspondence problem. For smartphone applications, the resulting 3D depth map can be used for further facial authentication. However, when using light in the NIR wavelength range, the interaction between the light and the display electronics causes a problem known as flicker.

[0003] Steiner et al., “Design of an Active Multispectral SWIR Camera System for Skin Detection and Face Verification”, JOURNAL OF SENSORS, Vol. 2016, January 1, 2016, pp. 1-16, XP055570013, US ISSN: 1687-725X, DOI: 10.1155 / 2016 / 9682453, relates to biometric facial recognition using spectral features in the short-wave infrared range of a material surface. The problem to be solved

[0004] Therefore, the object of the present invention is to provide devices and methods that address the aforementioned technical challenges posed by known devices and methods. Specifically, the object of the present invention is to provide methods and devices that allow for the reduction and / or prevention of flickering. Summary of the Invention

[0005] This problem is addressed by an imaging system having the features of the independent claims, an apparatus for authenticating a user of an authenticating device, a method for authenticating a user, a computer program, a computer-readable storage medium, a non-transient computer-readable medium, and uses thereof. Advantageous embodiments that can be implemented independently or in any arbitrary combination are set forth in the dependent claims and throughout the specification.

[0006] In a first aspect, an imaging system is disclosed. This imaging system includes...

[0007] - At least one pattern illumination source configured to emit at least one light pattern comprising a plurality of light beams having wavelengths in a short-wavelength infrared spectral region, wherein the wavelengths of the light are from 1000 nm to 1200 nm or from 1300 nm to 1500 nm; and

[0008] - At least one image sensor configured to generate at least one pattern image while the pattern illumination source emits the light pattern, wherein the image sensor is at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region.

[0009] As used herein, the term "system" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any set of interacting or interdependent components forming a whole. Specifically, these components may interact with each other to achieve at least one common function. At least two components may be processed independently, or may be coupled or connected.

[0010] As used herein, the term "imaging system" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, a system configured for imaging. The terms "imaging" and "generating at least one image" are used synonymously herein. As used herein, the term "imaging" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, capturing and / or determining and / or recording at least one image using an image sensor. Imaging may include capturing a single image and / or multiple images, such as an image sequence. The capture and / or determination and / or recording of images may be caused and / or initiated by the hardware and / or software interface in order to generate images via a hardware and / or software interface. For example, imaging may include continuously recording a sequence of images, such as video or film. Imaging may be initiated by user action or automatically, for example, upon automatic detection of the presence of at least one object or user within and / or a predetermined area of ​​the image sensor's field of view. As used herein, the term "field of view" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the angular range of the observable world and / or at least one scene that can be captured or viewed by an optical system (such as an image sensor). Field of view is typically expressed in degrees and / or radians, and exemplaryly may represent the total angle spanned by an image and / or the visible area.

[0011] As used herein, the term "image sensor" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one unit of a system configured to generate at least one image. The image may be generated via a hardware and / or software interface, which may be considered an image sensor. As used herein, the term "image" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, data recorded using an image sensor, such as multiple electronic readings from the image sensor. An image may include raw image data or may be a preprocessed image. For example, preprocessing may include applying at least one filter and / or at least one background correction and / or at least one background subtraction to the raw image data.

[0012] As used herein, the term "irradiation" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, the process of exposing at least one element to light. As used herein, the term "irradiation source" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, a device configured to generate at least one beam of light for irradiating an object. As used herein, the term "object" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, any target selected from living and non-living objects. An object may be or may include one or more organisms and / or one or more parts thereof, such as one or more body parts of a human (e.g., a user). An object may be a non-living object, such as a silicon mask or a printed image of a human. As used herein, the term "living organism" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term can specifically refer to, but is not limited to, any living being, particularly a living human being. As used herein, the term "living human being" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term can specifically refer to, but is not limited to, an individual of the Homo sapiens species, wherein the individual is currently alive.

[0013] As used herein, the term "pattern illumination source" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, an optical device configured to project at least one light pattern. As used herein, the term "projection" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the process of providing at least one light beam (particularly a light pattern) onto at least one surface.

[0014] As used herein, the term "light pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary pattern comprising multiple light spots. The light spots may extend at least partially in space. At least one light spot or any light spot may have any shape. In some cases, a circular shape of at least one light spot or any light spot may be preferred. The light pattern may include at least one dot pattern. The light pattern may be a coherent light pattern. The beam of the light pattern may have a single wavelength or multiple wavelengths, for example, to allow for additional measurements in other wavelength channels. The light pattern may include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further embossed patterns. For example, the light pattern is a hexagonal pattern, preferably a hexagonal light pattern, preferably a 2 / 5 hexagonal light pattern. Using a periodic 2 / 5 hexagonal pattern allows for the differentiation of artifacts from the available signal.

[0015] The emitted light pattern can illuminate the surface by comprising a light pattern of multiple light spots. The light spots can at least partially overlap. For example, the number of light spots can be equal to the number of light beams associated with the emitted light pattern. The intensities associated with the light spots can be substantially similar. "Substantially similar" can mean that the intensity values ​​associated with the light spots can differ by less than 50%, preferably less than 30%, and more preferably less than 20%. Using patterned light can be advantageous because it allows for the avoidance of photosensitive areas such as the eyes. The pattern can include at least one dot pattern.

[0016] A patterned illumination source may include at least one emitter, and in particular multiple emitters. As used herein, the term "emitter" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a particular or custom meaning. The term may specifically refer to, but is not limited to, at least one arbitrary device configured to provide at least one beam. The emitter may be selected from the group consisting of: at least one laser source; at least one laser diode; at least one vertical-cavity surface-emitting laser (VCSEL); at least one edge-emitting laser.

[0017] A patterned illumination source is configured to emit light having wavelengths within the short-wavelength infrared (SWIR) spectral region. The patterned illumination source can be configured to illuminate an object using a light pattern having a center wavelength within the SWIR. As used herein, the term "short-wavelength infrared spectral region" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or customary meaning. Specifically, the term may refer to light in the spectral region from 1000 nm to 3000 nm.

[0018] For example, the wavelength of light is from 1000 nm to 1200 nm, preferably from 1050 nm to 1150 nm, such as 1100 nm. In these wavelength regions, the reflective behavior of skin differs from that of other materials.

[0019] For example, the wavelength of light is from 1300 nm to 1500 nm, preferably from 1300 nm to 1400 nm. In these wavelength regions, the skin has distinctly different reflectance values.

[0020] Furthermore, the spot structure of the light spot on the object's surface is suitable for material detection in the aforementioned wavelength region, particularly using beam profiling analysis. Compared to shorter (NIR) wavelengths, the SWIR reflection and backscattering of materials associated with secure facial authentication (e.g., spoofing targets and human skin) can reveal additional distinguishing information in material comparisons. This can be used to better identify spoofing targets. Moreover, this wavelength can be advantageous when used in conjunction with an imaging system and a display. For wavelengths within the SWIR range, flicker effects or subtypes of such effects can be significantly reduced. Additionally, higher wavelengths impose fewer restrictions on eye safety, allowing for the use of higher laser power and / or longer illumination and exposure times. If used under a display, this can provide the advantage of compensating for lower display transmittance. It also provides more design freedom for projector specifications. Furthermore, for the aforementioned wavelength region, solar radiation is significantly lower compared to other wavelengths of SWIR.

[0021] The imaging system may include at least one optical element configured to modify the light spot generated by a patterned illumination source. Specifically, the optical element may be selected from the group consisting of at least one lens; at least one microlens array (MLA); at least one diffractive optical element (DOE); and at least one metasurface element. For example, an emitter may be used in combination with at least one optical element (such as an MLA, DOE, metasurface, or lens). The patterned illumination source may include at least one optical element configured to modify the light spot generated by the emitter, wherein the optical element is selected from the group consisting of at least one lens; at least one microlens array (MLA); at least one diffractive optical element (DOE); and at least one metasurface element. The optical element may be configured to generate multiple beams from a single incident beam. For example, the emitter may project up to 2000 light spots, and the optical element (e.g., including multiple metasurface elements) may be used to replicate the number of light spots. Additional arrangements (particularly including different numbers of projection emitters and / or at least one different optical element configured to increase the number of light spots) may be possible. Other multiplication factors are also possible.

[0022] The pattern illumination source includes at least one transmission device. As used herein, the term "transmission device," also referred to as "transmission system," is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, one or more optical elements suitable for modifying a beam, particularly for generating at least a portion of a light pattern, by modifying, for example, one or more of the beam parameters, beam width, or beam direction of the beam. The transmission device may include at least one imaging optical device. Specifically, the transmission device may include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one focusing lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one reflecting mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system; at least one holographic optical element; at least one meta-optical element. Specifically, the transmission device includes at least one stack of refractive optical lenses. The transmission device may include a multi-lens system with refractive properties.

[0023] The imaging system may further include at least one floodlight source configured to emit floodlight. As used herein, the term "floodlight source" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary device configured to provide substantially continuous spatial illumination. As used herein, the term "floodlight" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, substantially continuous spatial illumination, particularly diffuse and / or uniform illumination. A floodlight source may include at least one emitter, particularly multiple emitters. A floodlight source may include at least one LED or at least one VCSEL, preferably multiple VCSELs. The multiple VCSELs may overlap on a uniform area. As used herein, the term "substantially continuous spatial illumination" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, uniform spatial illumination, wherein non-uniform areas are possible. The area illuminated by a floodlight source, such as covering a user, a portion of the user, and / or the user's face, can be continuous. Power can be distributed across the entire illumination field. In contrast, illumination provided by a light pattern can include at least two continuous areas, particularly multiple continuous areas, and / or power can be concentrated in a smaller area of ​​the illumination field (compared to the entire illumination field). Floodlighting can be adapted to illuminate continuous areas, particularly a single continuous area. Patterned illumination can be adapted to illuminate at least two continuous areas.

[0024] A floodlight source can illuminate a measurement area, such as an object or a portion of an object, with a substantially constant intensity. As used herein, the term "constant" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the temporal aspect during the exposure time. Floodlight can vary over time and / or can be substantially constant over time. As used herein, the term "substantially constant" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, completely constant illumination, and embodiments that allow deviations from constant illumination of ≤ ± 10%, preferably ≤ ± 5%, more preferably ≤ ± 2%.

[0025] The center wavelength of the floodlight and the center wavelength of the light pattern can be the same, for example, within ±100 nm. However, other embodiments are conceivable.

[0026] The floodlight can have a wavelength in the short-wavelength infrared spectral region. Specifically, the wavelength of the floodlight can be from 1300 nm to 1500 nm, preferably from 1300 nm to 1400 nm.

[0027] Classic 2D spoofing targets are manufactured to resemble human faces and mimic the reflectivity of human skin in the visible light range. While using NIR (non-reflective optical character refraction) can enhance security, a range of common materials possess the same reflective properties as skin. Skin exhibits clearly distinguishable reflectivity values ​​for wavelengths within the SWIR region, making these wavelengths advantageous for differentiating human skin from spoofing targets. Therefore, these wavelengths can be beneficial for the authentication process. Furthermore, using these wavelength ranges is advantageous because light in these ranges is invisible to humans, and these wavelengths can be associated with low intensities in the solar spectrum. Thus, using the aforementioned wavelength range can provide the advantage of resistance to solar radiation.

[0028] The emission of the floodlight and the illumination of the light pattern can be performed subsequently or at least at a time that overlaps. For example, the floodlight and the light pattern can be emitted simultaneously. For example, one of the floodlight or the light pattern can be emitted at a lower intensity compared to the other.

[0029] As used herein, the term "light" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a particular or customary meaning. Specifically, the term may refer to, but is not limited to, electromagnetic radiation in one or more of the infrared, visible, and ultraviolet spectral ranges. In this document, the term "ultraviolet spectral range" generally refers to electromagnetic radiation with wavelengths from 1 nm to 380 nm, preferably from 100 nm to 380 nm. Further, in part according to the standard ISO-21348, the effective version of this document as of the date of this document, the term "visible spectral range" generally refers to the spectral range from 380 nm to 760 nm. The term "infrared spectral range" (IR) generally refers to electromagnetic radiation from 760 nm to 1000 µm, wherein the range from 760 nm to 1.5 µm is generally referred to as the "near-infrared spectral range" (NIR), the range from 1.5 µm to 15 µm is referred to as the "mid-infrared spectral range" (MidIR), and the range from 15 µm to 1000 µm is referred to as the "far-infrared spectral range" (FIR).

[0030] As used herein, the term "ray" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a line perpendicular to the wavefront of light and pointing in the direction of energy flow. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a collection of rays. In the following text, the terms "ray" and "beam" will be used as synonyms. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a quantity of light, specifically a quantity of light traveling substantially in the same direction, including the possibility that the beam has an extension angle or widening angle.

[0031] As mentioned above, using wavelengths within the SWIR spectrum has several advantages. However, detectors typically used in authentication systems quickly become inefficient for these wavelength ranges, making the use of higher wavelengths problematic. Image sensors are at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region. Image sensors may include at least one quantum dot sensor. For example, a quantum dot sensor may be designed as described in US 2023 / 258498A1. Image sensors may include a substrate made of, for example, a semiconductor material (e.g., silicon). The substrate may be made of an insulating layer (e.g., made of silicon oxide) and conductive traces and vias (e.g., made of copper). The substrate may be covered by the insulating layer. The insulating layer may be made of silicon oxide. Conductive vias may extend through the insulating layer. The vias may be made of copper or tungsten. Image sensors may include at least one via for each pixel. The location of each pixel may position a via. The via for each pixel may correspond to an electrode of the pixel. Image sensors may further include a layer covering the insulating layer and all vias corresponding to the pixels. The layer may be continuous at all pixel locations. The layer may contact the vias. The layer may be made of quantum dots. Quantum dots can be fixed together with a resin or matrix and attached to an insulating layer. Preferably, the quantum dot layer may contain only quantum dots and resin. All quantum dots in the layer may be substantially identical. All quantum dots in the quantum dot layer have substantially the same size and are made of the same composition. For example, all quantum dots in the quantum dot layer are composed of lead sulfide (PbS). Quantum dots, or semiconductor nanoparticles, are nanomaterial structures that generate electron-hole pairs when a given photon is incident on the nanomaterial structure. In this way, for example, detection elements, such as photodetectors, can be generated based on semiconductor nanoparticles. Quantum dots include a semiconductor core. Quantum dots may also include a shell surrounding the core to protect and passivate it, preferably made of a semiconductor material. Quantum dots further include ligands, organolithic compounds, organometallic compounds, or inorganic molecules extending from the shell and passivating, protecting, and functionalizing the semiconductor surface. The composition of quantum dots may be selected from the following materials. The core is made of, for example, materials or alloys of the following: CdSe, CdS, CdTe, CdSeS, CdTeSe, AgS, ZnO, ZnS, ZnSe, CuInS, CuInSe, CuInGaS, CuInGaSe, PbS, PbSe, PbSeS, PbTe, InAsSb, InAs, InSb, InGaAs, InP, InGaP, InAlP, InGaAlP, InZnS, InZnSe, InZnSeS, HgTe, HgSe, HgSeTe, Ge, Si.The shell is made of, for example, materials or alloys of the following: CdSe, CdS, CdTe, CdSeS, CdTeSe, AgS, ZnO, ZnS, ZnSe, CuInS, CuInSe, CuInGaS, CuInGaSe, PbS, PbSe, PbSeS, PbTe, InAsSb, InAs, InSb, InGaAs, InP, InGaP, InAlP, InGaAlP, InZnS, InZnSe, InZnSeS, HgTe, HgSe, HgSeTe, Ge, Si.

[0032] Using quantum dot image sensors allows for efficient image detection even in SWIR.

[0033] As described above, for example, the wavelength of the light is from 1050 nm to 1150 nm, preferably 1100 nm, or the wavelength of the light is from 1300 nm to 1400 nm. The quantum efficiency of the image sensor can be >10%, preferably >20%, and more preferably >40% in the short-wavelength infrared spectral region.

[0034] As used herein, the term "pattern image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an image generated by an image sensor while an object is illuminated by a pattern of light. A pattern image may include an image showing at least a portion of an object, particularly the user's face, when illuminated by the pattern, particularly over a corresponding region of interest included in the image. Thus, the image sensor can generate an intensity image of the object projected using the pattern. A pattern image can be generated by imaging and / or recording the light reflected from the object illuminated by the pattern of light.

[0035] An imaging system may be included in the device. The device may include a display.

[0036] As used herein, the term "display" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a device of any shape configured for displaying information. This information can be any type of information, such as at least one image, at least one chart, at least one histogram, at least one graphic, text, numbers, at least one symbol, operation menus, etc. A display may be or may include at least one display panel. A display may have any shape, such as a rectangular shape. A display may be the front display of a device. A display may include at least one of a display panel (particularly comprising multiple pixels and / or multiple transistors) or glass (specifically, a covering glass, particularly configured to cover the display panel).

[0037] As used herein, the term "cover glass" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, elements made of glass material configured to cover a display panel. Cover glass may also be referred to as protective glass or front glass. Cover glass can be configured to protect the display panel, particularly from environmental influences such as mechanical effects. Cover glass can be made of glass material with a refractive index between 1.46 and 1.9 (at 940 nm). For example, the glass material may be crown glass with a refractive index of 1.5 (at 940 nm). For example, the glass material may be borosilicate glass with a refractive index of 1.51 (at 940 nm).

[0038] The display, specifically the display panel, may be or may include at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED). As used herein, the term "organic light-emitting diode" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, a light-emitting diode (LED), wherein the emitting electroluminescent layer is an organic compound film configured to emit light in response to an electric current. OLED displays may be configured to emit visible light. As used herein, the term "organic light-emitting diode" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, display technologies that utilize semiconductor particles called quantum dots to generate colors on a display. These quantum dots, when excited by light, can emit a variety of different colors of light depending on their size. By using combinations of red, green, and / or blue quantum dots, QLED displays can display a variety of colors with high brightness and color accuracy.

[0039] An image sensor and / or pattern illumination source may be arranged behind the display. Light can pass through the display when illuminated from the illumination source. The display may be at least partially transparent. The display may be at least partially transparent in at least one continuous area covering the pattern illumination source and / or floodlight source and / or image sensor. For example, the display may include perforations covering a continuous area of ​​the pattern illumination source, floodlight source, and / or image sensor. For example, the display may have a transmittance of less than or equal to 20%, preferably less than or equal to 15%, more preferably less than or equal to 10%. For example, the intensity of the light beam after it has passed through the display may correspond to ≤ 10% of the associated intensity when the light beam was emitted.

[0040] The display may be at least partially transparent in at least one continuous area covering the image sensor. The display may be at least partially transparent in at least one continuous area in a manner such that at least one of the following is true:

[0041] - The light pattern incident on a continuous area passes through the display while being illuminated from the projector;

[0042] - Floodlight incident on a continuous area passes through the display while being illuminated from the floodlight source;

[0043] User light, generated by the pattern of light incident on the user and / or floodlight, is incident on a continuous area and passes through the display to strike the image sensor.

[0044] As used herein, the term "at least partially transparent" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the property of a display to allow light, particularly light of a specific wavelength range (e.g., light in the infrared spectral region, particularly light in the near-infrared spectral region), to pass through at least partially. For example, a display may be translucent in the near-infrared region. For example, a display may have 20% to 50% transparency in the near-infrared region. A display may have different transparency for different wavelength ranges. The present invention may provide a device comprising an image sensor and an illumination source that can be placed behind the display of the device. The transparent area of ​​the display may allow the operation of the image sensor and projector behind the display.

[0045] As described above, the display can be at least partially transparent. A continuous, partially transparent area of ​​the display can be associated with a first pixel density value (pixels per inch (PPI)), and another area of ​​the display can be associated with a second pixel density value. The first pixel density value can be lower than the second pixel density value. Light transmission through the continuous area can be higher than transmission through the other area. The first pixel density value can be equal to or lower than 450 PPI, preferably between 300 and 440 PPI, more preferably between 350 and 450 PPI. The first pixel density value can remain constant over the entire continuous area, with a maximum deviation of 20% or preferably 10%. The second pixel density value can be between 400 and 500 PPI, preferably between 450 and 500 PPI.

[0046] A continuous, at least partially transparent region of the display may include a first region and a second region. The first region may be associated with a first number of transistors configured to control at least one pixel, and the second region may be associated with a second number of transistors configured to control at least one pixel, wherein the first number of transistors may be less than the second number of transistors. The first number of transistors and / or the second number of transistors may refer to, or be, the density of these transistors.

[0047] As used herein, the term "pixel" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a picture unit representing an addressable element, particularly the smallest picture unit. The collective of these pixels may represent a display. Pixels can be manipulated by changing their color, brightness, and / or contrast, etc. In particular, for the purpose of manipulating a pixel, it may be driven by at least one transistor, exemplarily by a transistor that controls the current required to drive the pixel. Typically, thin-film transistors (TFTs) can be used to drive pixels. TFTs are preferably used in flat panel displays.

[0048] For example, the device is selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0049] On the other hand, the use of an imaging system according to the invention (such as that described in one or more embodiments above or in more detail below) for authenticating a user of a device including the imaging system is disclosed. Regarding embodiments and definitions, reference is made to the description of the imaging system above and the devices and methods for authentication described in more detail below.

[0050] On the other hand, a device for authenticating users of a device to perform at least one operation on the device that requires authentication is disclosed.

[0051] The equipment used for authentication includes

[0052] -Imaging systems according to the invention, such as those described in one or more embodiments above or described in more detail below;

[0053] - At least one display, wherein the image sensor is disposed behind the display, wherein the display is at least partially transparent in at least one continuous area covering the image sensor.

[0054] - At least one authentication unit, which is configured to use the pattern image to perform at least one authentication process for a user.

[0055] For examples and definitions, refer to the description of the imaging system above and the devices and methods for authentication described in more detail below.

[0056] As used herein, the term "user" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a person who intends to use the device and / or uses the device.

[0057] The device can be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0058] As used herein, the term "authentication" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, verifying the identity of a user. Specifically, authentication may include distinguishing a user from other humans or objects, particularly distinguishing authorized access from unauthorized access.

[0059] Authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, such as providing it to other devices or units (e.g., to at least one authorized unit) to authorize access to the device. The identity information can be proven through authentication. For example, the identity information may be and / or may include at least one identity token. In the event of successful authentication, the facial image recorded by the imaging detector (such as the imaging sensor described above) can be verified as an image of the user's face, and / or the user's identity is verified.

[0060] Authentication can be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one face detection (e.g., on at least one floodlight image, as will be described in more detail below), and at least one recognition step, wherein an identity is assigned to the detected face and / or at least one identity check and / or verification of the user's identity is performed.

[0061] Authentication can be and / or may include biometric authentication. As used herein, the term "biometric authentication" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, authentication using at least one biometric identifier (e.g., a unique, measurable characteristic used to identify and describe an individual). A biometric identifier may be a physiological characteristic.

[0062] The operation requiring authentication on the device can be any operation that requires access to at least one resource associated with the device. The method may include receiving a request for access to at least one resource associated with the device and performing at least one authentication process.

[0063] As used herein, the term "access" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, accessing and / or using one or more functions associated with a device. As used herein, the term "function associated with a device" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any function, such as accessing at least one element and / or at least one resource of the device or associated with it. Functions requiring user authentication may be predefined. One or more functions associated with a device may include unlocking the device and / or accessing an application preferably associated with the device and / or accessing a portion of an application preferably associated with the device. For example, the function may include accessing content of the device, such as content stored in the device's database and / or content that can be retrieved by the device. In embodiments, allowing a user to access a resource may include allowing a user to perform at least one operation with the device and / or system. A resource may be a device, a system, a function of the device, a function of the system, and / or an entity. Additionally and / or alternatively, allowing a user to access a resource may include allowing a user to access an entity. An entity may be a physical entity and / or a virtual entity. Virtual entities can be, for example, databases. Physical entities can be restricted areas. Restricted areas can be one of the following: secure areas, rooms, apartments, vehicles, portions of the examples mentioned earlier, etc. Devices may be locked and can only be unlocked by authorized users.

[0064] The device may further include at least one communication interface, such as a user interface, which is configured to receive requests for accessing at least one resource associated with the device, particularly for performing at least one operation requiring authentication on the device.

[0065] As used herein, the term "communication interface" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an article or element forming a boundary configured for transmitting information. In particular, a communication interface may be configured to transmit information from a computing device (e.g., a computer), such as to send or output information to, for example, another device. Alternatively or additionally, a communication interface may be configured to transmit information to a computing device, such as to a computer, such as to receive information. A communication interface may specifically provide a means for transmitting or exchanging information. In particular, a communication interface may provide a data transmission connection, such as Bluetooth, NFC, Ethernet, inductive coupling, etc. As an example, a communication interface may be or may include at least one port, including one or more of a network or internet port, a USB port, and a disk drive. A communication interface may be at least one network interface. As used herein, the term "user interface" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term can refer to, but is not limited to, features of a device configured to interact with its environment (e.g., for the purpose of exchanging information in one or more directions, such as exchanging data or commands). For example, a user interface can be configured to share information with a user and receive information provided by the user. A user interface can be a feature that interacts with the user visually (e.g., a display) or with the user auditorily. As an example, a user interface can include one or more of the following: a graphical user interface; a data interface, such as a wireless and / or wired data interface.

[0066] As used herein, the term "request for access" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, at least one action and / or instance of requesting access. As used herein, the term "receive request" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, the process of obtaining a request from, for example, a data source and / or a user interface. Receiving may be fully or partially automated. Receiving a request for access to one or more functions associated with a device can be performed using at least one communication interface. Receiving may include, for example, receiving at least one user input via at least one user interface (e.g., such as the device's display), and / or receiving a request from a remote device and / or the cloud (e.g., via device communication, such as via the Internet). For example, a request may be generated or triggered by at least one user input (e.g., via entering a security number or other unlocking action by the user), and / or may be sent from a remote device and / or the cloud (e.g., via a connected account).

[0067] The authentication process can be executed using at least one authentication unit configured to perform at least one authentication procedure for a user. An authentication unit may include at least one processor. Execution of the authentication process can be triggered and / or started by receiving a request.

[0068] An authentication unit may include at least one processor. As is generally used herein, the term "processor" (also referred to as a processing unit) is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any logic circuit configured to perform basic operations of a computer or system, and / or generally to a device configured to perform computations or logical operations. In particular, a processor may be configured to process the basic instructions that drive a computer or system. As an example, a processor may include at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU) (such as a math coprocessor or a number coprocessor), multiple registers (specifically configured to provide operands to the ALU and store the results of operations), and memory (such as L1 and L2 cache memories, etc.). In particular, a processor may be a multi-core processor. Specifically, a processor may be or may include a central processing unit (CPU). Additionally or alternatively, a processor may be or may include a microprocessor; therefore, specifically, the elements of a processor may be contained within a single integrated circuit (IC) chip. Alternatively or additionally, the processor may be or may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or one or more chips, such as dedicated machine learning optimization chips. Specifically, the processor may be configured, for example, by software programming, to perform one or more evaluation operations. At least one or any component of the computer program configured to perform the authentication process may be executed by the processing device. Alternatively or additionally, the authentication unit may be or may include a connectivity interface. The connectivity interface may be configured to transfer data from one device to a remote device; or vice versa. At least one or any component of the computer program configured to perform the authentication process may be executed by a remote device.

[0069] The authentication process may include multiple steps. For example, the authentication process may include performing at least one face detection step. The face detection step may include analyzing at least one image of the user, generated, for example, by an image sensor or another camera. The image may be a floodlight image. As used herein, the term "floodlight image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, an image generated by an image sensor when an illumination source emits infrared floodlight (e.g., on an object and / or user). The floodlight image may include an image showing the user, particularly the user's face, while the user is being illuminated with floodlight. The floodlight image can be generated by imaging and / or recording the light reflected by the illuminated object and / or user. The floodlight image showing the user may include at least a portion of the floodlight on at least a part of the user. For example, the illumination and imaging of the floodlight source may be synchronized, for example, by using at least one control unit.

[0070] Determining whether a user corresponds to an authorized user can be performed using an image of the user generated simultaneously with SWIR illumination of the object. This allows the imaging to be undetectable to humans and to a very small extent included within the solar spectrum. The authentication process may include generating at least one floodlight image showing the associated user simultaneously with floodlight illumination, and determining, based on the floodlight image, whether the user's identity corresponds to a verified identity. The authentication process may include allowing the user to access the resource if their identity corresponds to a verified identity, and otherwise denying the user access to the resource if their identity does not correspond to a verified identity.

[0071] The certification process may include:

[0072] -Utilize floodlight illumination on users by using at least one floodlight source;

[0073] - The at least one floodlight image is captured by using at least one image sensor or another camera of the device.

[0074] The face detection step may include analyzing a floodlight image. For example, an authentication process may include performing at least one face detection using a floodlight image. Face detection can be performed locally on the device. However, face recognition (i.e., assigning an identity to a detected face) can be performed remotely, for example, in the cloud, especially when identification rather than just verification is required. User templates can be stored at a remote device, such as in the cloud, and do not need to be stored locally. This can be advantageous from a storage and security perspective.

[0075] The authentication process may include identifying a user based on a floodlight image. As used herein, the term "identification" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, assigning identity to a detected face and / or at least one identity check and / or verification of the user's identity. Therefore, in particular, the authentication unit may forward data to a remote device. Alternatively or additionally, the authentication unit may perform user identification based on a floodlight image, particularly by running an appropriate computer program with corresponding functionality.

[0076] Recognition may include assigning an identity to a detected face and / or verifying the user's identity. Recognition may include performing facial verification on the imaged face to confirm whether it is the user's face. Identifying the user may include matching a floodlight image (e.g., showing the outline of parts of the user, particularly parts of the user's face) against a template. To match the floodlight image with the template, the similarity between at least one image feature vector obtained from the floodlight image and at least one template feature vector may be considered and / or evaluated. This template vector may be obtained from a template image.

[0077] The template image can be generated during the registration process. As used herein, the term "registration process" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, at least one step of registration, particularly registration with the service. During the registration process, the template image can be generated under secure conditions, in a manner that ensures the generated template image is presented to the user. The registration process may include at least one of the following steps: capturing the template image; recording personal data, etc.

[0078] Face detection may include analyzing floodlight images. Specifically, the analysis of floodlight images may include using at least one image recognition technique, particularly face recognition technology. Image recognition technology includes at least one process for identifying a user in an image. Image recognition may include using at least one technique selected from the following: color-based image recognition, for example using features such as template matching; segmentation and / or connected component (blob) analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0079] For example, authentication may include identifying a user. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing facial verification on the imaged face to confirm whether it is the user's face. Identifying a user may include matching a flood image (e.g., showing the outlines of various parts of the user, particularly the outlines of various parts of the user's face) with a template (e.g., a template image generated during registration). Identifying a user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the flood image can match the image template, authentication may be successful. If the flood image cannot match the image template, authentication may be unsuccessful.

[0080] As used herein, the term "memory" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one electronic storage space configured to store data, instructions, and programs. The stored data, instructions, and / or programs can be forwarded to a processor for processing. Memory can be or may include at least one of the following: random access memory; read-only memory; cache memory; hard disk drive; solid-state drive; virtual memory.

[0081] To determine whether a user's identity corresponds to a verified identity based on a floodlight image, the similarity between at least one image feature vector obtained from the floodlight image and at least one template feature vector can be considered. This template vector can be obtained from a template image, which can be generated during the registration process.

[0082] For example, user identification may include determining multiple facial features. Analysis may include comparing the determined facial features with template features, specifically performing a matching process. Template features may be features extracted from at least one template. The template may be or may include at least one image generated during the registration process (e.g., when initializing the device). The template may be an image of an authorized user. Template features and / or facial features may include vectors. Feature matching may include determining the distance between vectors. User identification may include comparing the distance between the vectors with at least one predefined limit. If the distance is at least within tolerance and ≤ the predefined limit, the user can be successfully identified. Otherwise, the user may be rejected and / or refused.

[0083] Analysis of a flood image may further include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposing into color channels; decomposing into hue, saturation, and brightness channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian Gaussian filter; applying a difference Gaussian filter; applying the Sobel operator; applying the Laplacian operator; applying the Scharr operator; applying the Prewitt operator; applying the Roberts operator; applying the Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying the Radon transform; applying the Hough transform; applying the wavelet transform; thresholding; and creating a binary image. The region of interest may be manually determined by the user or automatically determined, for example, by identifying the user within the image.

[0084] For example, image recognition may include a trained model using at least one model, particularly one including at least one face recognition model. Analysis of floodlight images can be performed using a face recognition system such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in the following literature: MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for face recognition systems, please refer to: Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering”, arXiv:1503.03832. Labeled image data from image databases can be used as training data.Specifically, labeled faces can be used from one or more of the following sources: GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts Amherst, October 2007; the YouTube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset. Training of convolutional neural networks can be described as in "FaceNet: A Unified Embedding for Face Recognition and Clustering" by Florian Schroff, Dmitry Kalenichenko, and James Philbin, arXiv:1503.03832.

[0085] Determining whether a user corresponds to an authorized user may include using an image of the user generated when the user is illuminated by RGB light (e.g., sunlight and / or ambient light can be used, and existing hardware can be utilized without emitting further light). For example, in a smartphone, a selfie camera may be used. The method may include illuminating the user with RGB light, and simultaneously generating at least one RGB image showing at least a portion of the user using an image sensor. The method may further include using the RGB image to identify the user. To identify the user, the similarity between at least one image feature vector obtained from the RGB image and at least one template feature vector may be considered. The template feature vector can be obtained from a template image. This template image may be generated during the registration process. User authentication may include facial authentication. The feature vector is a facial feature vector. The template image satisfies one or more of the following: shows, includes, or represents the face of an authorized user. For a description of analyzing and identifying users using floodlight images, refer to the description of analyzing and identifying users.

[0086] The authentication unit is configured to perform at least one authentication process for a user using a pattern image. The authentication unit may be configured to extract liveness data from the pattern image. The authentication unit may be configured to use the liveness data to allow or deny the user from performing at least one authentication-required operation on the device.

[0087] As used herein, the term "living data" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, data that provides indications that an object corresponds to a living organism. In particular, extracting living data includes extracting material data and / or extracting blood perfusion data.

[0088] As used herein, the term "material data" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or customary meaning. The term may specifically refer to, but is not limited to, data providing information about materials. Material data may include information about the material type of an object's surface under illumination, for example, by a pattern of light. Extracting material data may be or may include generating data about the material type and / or deriving data from the material type. Material data may include information about the material type of an object. Advantageously, material testing must only be effective on targets with reflectivity similar to human skin.

[0089] Material data can be extracted from a pattern image. The material data can indicate the type of material. In particular, the material data can indicate whether an object associated with the image at least partially comprises skin. The material data can be associated with an object, especially with an object shown in the image. Extracting material data from a pattern image can include generating material type and / or data derived from the material type.

[0090] Material data extraction can be based on pattern images, more preferably on one or more partial images. For example, the image can be reduced to a predefined size by applying one or more image processing techniques. Reduction may include selecting at least one region of interest and segmenting the pattern image into regions of the predefined size pattern image. The regions of the predefined size image may be associated with an object. Image processing techniques may include at least one image recognition technique, particularly facial recognition technique. Image recognition techniques include at least one process for identifying a user in an image. Image recognition may include at least one technique using techniques selected from the following: color-based image recognition, such as using features like template matching; segmentation and / or connected component (blob) analysis, such as using size or shape; machine learning and / or deep learning, such as using at least one convolutional neural network. Parts of the image other than the regions of the predefined size image may be associated with the background and / or may be unrelated to the object. Image portions useful for subsequent analysis can be selected.

[0091] Material data may include at least one measure of reflectivity.

[0092] Extracting material data from patterned images can include beam profile analysis of light spots. For more information on beam profile analysis, see WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference. Beam profile analysis can allow for reliable classification of objects based on several light spots. Each light spot in a patterned image can include a beam profile. As used herein, the term "beam profile" generally refers to at least one intensity distribution of a light spot on an optical sensor as a function of pixels. Beam profiles can be selected from the group consisting of: trapezoidal beam profiles; triangular beam profiles; conical beam profiles; and linear combinations of Gaussian beam profiles.

[0093] To extract material data, a complete pattern image can be used. Alternatively, a partial image can be used. Material data extraction may include generating one or more partial images from a pattern image. For example, different regions of the pattern image can be selected as partial images. The partial images can be different from each other. In particular, the partial images can be non-overlapping. Using partial images allows for the collection of material data from different regions of an object (e.g., under different lighting conditions) and / or comparison of the extracted material data and / or generation of material maps and / or reduction of uncertainty in the obtained material data.

[0094] At least one algorithm can be used to evaluate material data (e.g., reflectivity), where the algorithm is designed to check whether the object is a deceptive target. Alternatively, artificial neural networks can be used to check whether an image is a deceptive target.

[0095] Material data can be determined by providing an image to a model and receiving material from the model. Material data can be extracted using at least one model. Extracting material data can include providing an image to at least one model and / or receiving material data from the model. Providing an image to the model can include, and subsequently may be, receiving the image at the model's input layer or via a model loss function.

[0096] The model can be a data-driven model. Data-driven models can include convolutional neural networks and / or encoder-decoder structures, such as autoencoders. Other examples for generating representations include FFT, wavelets, deep learning (such as CNNs), energy models, normalized flow, GANs, visual transformers or transformers for natural language processing, and autoregressive image modeling. Supervised or unsupervised schemes can be applied to generating representations and also to generating embeddings in ML languages, such as cosine or Euclidean metrics.

[0097] The model can be a data-driven model. Data-driven models can include convolutional neural networks and / or encoder-decoder structures, such as autoencoders. Other examples for generating representations include FFT, wavelets, deep learning (such as CNNs), energy models, normalized flow, GANs, visual transformers or transformers for natural language processing, and autoregressive image modeling. Supervised or unsupervised schemes can be applied to generating representations and also to generating embeddings in ML languages, such as cosine or Euclidean metrics.

[0098] Comparing the extracted material data with expected material data, particularly skin material data, can include determining whether the material data matches the expected material data, at least within tolerance. In this case, the object's surface is classified as skin. Otherwise, if the material data does not match the expected material data, the object is determined to correspond to a non-living organism.

[0099] Determining whether a surface is human skin may include comparing extracted material data with skin-related material data (skin material data). Comparing the material data with desired material data may include determining the similarity between the extracted material data and the skin material data. Skin material data may refer to predetermined material data of skin. For example, material data may include reflectance values, and the method may include comparing the determined reflectance values ​​of the surface with at least one range of reflectance values ​​of skin. The method may include considering tolerances such as ±10%, preferably ±5%, more preferably ±1%.

[0100] Skin material data may be stored in at least one database and / or may be retrieved from at least one database, for example, the database may be at least partially cloud-based, for example, via at least one communication interface.

[0101] An authentication unit can be configured to perform human skin detection using a pattern image. If the material data matches the material data of the skin, the object is determined to correspond to a human. This allows for the determination of whether the object corresponds to a living organism. Otherwise, if the material data does not match the material data of the skin, the object is determined to correspond to a non-living organism. The comparison of the material data with the skin material data can lead to allowing and / or denying the user and / or object from performing at least one operation requiring authentication. The authentication process can include allowing the user to access the resource if it is determined that the user's pattern image corresponds to a living organism, and otherwise denying the user access to the resource if it is determined that the user's pattern image does not correspond to a living organism. For example, the authentication process can be verified based on the extracted material data. Verification can include determining the similarity between the extracted material data and skin, such as comparing the extracted material data with skin material data. The comparison of the material data with the skin can lead to allowing and / or denying the user and / or object from performing at least one operation requiring authentication. In an example, skin can be compared as the desired material data with a non-skin material or silicon as the material data, and the result could be rejection because silicon or non-skin materials may differ from skin.

[0102] The authentication unit may be further configured to consider additional security features, such as those extracted from the pattern image. Specifically, the authentication unit may be further configured to extract live data (e.g., blood perfusion measurements) and / or consider live data extracted from the pattern image. As used herein, the term "blood perfusion measurement" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, blood flow through a given volume or mass of tissue. Typically, blood perfusion measurement may be given in ml / ml / s or ml / 100 g / min. Blood perfusion measurement may represent localized blood flow through at least one capillary network and one or more extracellular spaces in body tissue. Determining the at least one blood perfusion measurement may include determining at least one speckle contrast of the pattern image. Alternatively or additionally, determining the at least one blood perfusion measurement may include determining the blood perfusion measurement based on the determined at least one speckle contrast. As used herein, the term "speckle contrast" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the degree of variation in a speckle pattern generated by coherent light. The speckle pattern can be generated by a transmitter, particularly on an object. The speckle contrast can be a measure of the average contrast of the intensity distribution within a region of the speckle pattern. In particular, the speckle contrast K over a region of the speckle pattern can be expressed as the standard deviation σ versus the average speckle intensity. The ratio, that is,

[0103]

[0104] The speckle contrast can include speckle contrast values. These values ​​can range from 0 to 1. Blood perfusion measurement can be determined based on the speckle contrast. The blood perfusion measurement can depend on the determined speckle contrast. If the speckle contrast changes, the blood perfusion measurement obtained from the speckle contrast will also change accordingly. The blood perfusion measurement can be a single number or value that can represent the likelihood that the object is alive. To monitor changes in speckle contrast, multiple pattern images generated at different time points can be used. To determine the speckle contrast, a complete pattern image can be used. Alternatively, a portion of the pattern image can be used to determine the speckle contrast. Preferably, a portion of the pattern image represents a smaller region of the pattern image than the region of the complete pattern image. In an embodiment, a data-driven model can be used to determine the blood perfusion measurement. The data-driven model is parameterized and / or trained based on a training dataset. The training dataset can include pattern images and blood perfusion measurements. The data-driven model can be parameterized and / or trained based on the training dataset to output blood perfusion measurements based on received pattern images. In an embodiment, determining whether an object corresponds to a living organism based on blood perfusion data may include determining whether a blood perfusion measurement corresponds to a human blood perfusion measurement. Determining whether a blood perfusion measurement corresponds to a human may include comparing the blood perfusion measurement to at least one predefined or predetermined range of blood perfusion measurement values, for example, stored in at least one database. If the extracted blood perfusion measurement falls within the redefined or predetermined range of blood perfusion measurement values, at least within tolerance, the object is determined to correspond to a living organism; otherwise, it does not.

[0105] The device may further include at least one distance sensing system configured to determine the distance between an object and an image sensor. For example, an imaging system may be configured to provide a distance estimate. Alternatively or additionally, the device may include supplementary distance sensing systems, such as a triangulation system, at least one time-of-flight system, etc. The estimated distance can be considered to calculate reflectivity. The distance can be considered a corrected value for the reflectivity measure. This distance can be used to determine whether the distance between the object and the image sensor is within the working range of the model used to extract material data. Distance is also important for facial authentication because the facial recognition model is associated with a working range. This working range can specify the range of distances between the object and the image sensor, where the model operates and / or is trained on the user's image. In embodiments, the working range can specify at least one upper boundary and / or at least one lower boundary of the distance between the object and the image sensor and / or the illumination source. The working range can be associated with an authentication process. The working range can include at least one value. This value can be numerical, particularly a positive value. An indication of the working range can be received, particularly before determining whether the distance is within or outside the working range of the authentication process. The indication of the work scope can be used to determine whether a distance is within or outside the work scope of the certification process. The indication of the work scope can also be used to compare a distance to the work scope.

[0106] Distance can be calculated using at least one distance determination technique. For example, distance can be determined using one or more of the following: beam profiling analysis (e.g., as described in WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference), time-of-flight, triangulation, etc. The distance can be calculated using a distance sensing system of a mobile device, which includes an illumination source and an image sensor.

[0107] The authentication process may further include determining the user's distance information and determining whether the user is within or outside the scope of the authentication process by comparing the distance to the working area. The authentication process may include allowing the user to access resources if it is determined that the user is within the working area, and otherwise denying the user access to resources if it is determined that the user is outside the working area.

[0108] The authentication unit can be further configured to determine a depth map. The determined depth map can be compared to a predetermined depth map of the user, for example, determined during the registration process. The authentication unit can be configured to authenticate the user if the determined depth map matches (in particular, at least within tolerance) the user's predetermined depth map. Otherwise, the user can be rejected.

[0109] The authentication unit can forward data to a remote device. The authentication unit can be configured to outsource at least one step of the authentication process (e.g., user identification) and / or at least one step of the verification process (e.g., consideration of material data) to a remote device, specifically a server and / or a cloud server. This device and the remote device can be part of a computer network, particularly the Internet. Thus, the device can function as a field device used by the user to generate data required in the authentication process and / or its verification. The device can transmit the generated data and / or data associated with intermediate steps of the authentication process and / or its verification to the remote device. In this scenario, the authentication unit can be and / or may include a connection interface configured to transmit information to the remote device. Data generated by the remote device used in the authentication process and / or its verification can be further transmitted to the device. This data can be received by the connection interface included in the device. The connection interface can be specifically configured to transmit or exchange information. In particular, the connection interface can provide a data transmission connection. As an example, the connection interface can be or may include at least one port, including one or more of a network or Internet port, a USB port, and a disk drive.

[0110] It is important to emphasize that data from a device can be transferred to a specific remote device based on at least one circumstance (e.g., date, day, load of a particular remote device, etc.). A field device may not be able to select a specific remote device. Conversely, another device may choose which specific remote device the data can be transferred to. The authentication process and / or the generation of verification data may involve several different entities using the remote device. At least one entity may generate intermediate data and transfer that intermediate data to at least one other entity.

[0111] Allowing a user to access resources may include authorizing the user. The device may include at least one authorization unit configured to allow the user to perform at least one operation on the device, such as unlocking the device, upon successful authentication, or to deny the user from performing at least one operation on the device if authentication fails. Thus, the user is aware of the authentication result. The authorization unit may be configured to allow or deny the user to perform at least one operation requiring authentication on the device based on material data and identification, such as using floodlight images. The authorization unit may be configured to allow or deny the user access to one or more functions associated with the device, depending on authentication or denial. Allowing may include granting permission to access the one or more functions. The authorization unit may be configured to determine whether the user corresponds to an authorized user, wherein allowing or denying is further based on determining whether the user corresponds to an authorized user. As used herein, the term "authorization" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, the process of assigning access rights to a user (particularly selective permission or selective restriction of access to the device and / or at least one resource of the device). The authorization unit may be configured for access control. As used herein, the term "authorization unit" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a unit configured to authorize a user, such as a processor. An authorization unit may include at least one processor or may be designed as software or an application. The authorization unit and the authentication unit may be integrated, for example, by using the same processor. The authorization unit may be configured to allow a user to access, for example, one or more functions on the device, such as unlocking the device, upon successful authentication, or to deny the user access to, for example, one or more functions on the device, upon unsuccessful authentication.

[0112] For example, by using a user interface (such as the device's display), the device can be configured to display the results of authentication and / or authorization.

[0113] On the other hand, the use of a device for authenticating a user according to the present invention (such as those described in one or more embodiments above or in more detail below) for one or more of the following: in-vehicle payment, for vehicle access and / or for starting the vehicle; access control, such as for at least one resource of a mobile device like a mobile phone; in-cabin sensing; for payment, using a mobile phone and / or a fixed device, such as an ATM, for settlement. Regarding embodiments and definitions, refer to the description of the device above and the more detailed description of the device and method for authentication below.

[0114] On the other hand, a computer-implemented method for authenticating a user of a device to perform at least one operation requiring authentication on the device is disclosed. This method uses a device for authenticating a user according to the invention, such as those described in one or more embodiments above or in more detail below. Therefore, regarding embodiments and definitions, reference is made to the description of the device above and the device and method for authentication in more detail below.

[0115] The method includes:

[0116] - Triggers the illumination of the user with a light pattern from the pattern illumination source;

[0117] - While illuminating the user with the light pattern, trigger the generation of at least one pattern image showing at least a portion of the user using the image sensor;

[0118] - Trigger the extraction of liveness data from the pattern image;

[0119] - Use the liveness data to allow or deny the user to perform at least one action on the device that requires authentication.

[0120] These method steps can be executed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be executed repeatedly.

[0121] As used herein, the term "computer-implemented" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, methods involving at least one computer and / or at least one computer network. The computer and / or computer network may include at least one processor configured to perform at least one method step of the method according to the invention. Specifically, each of these method steps is performed via a computer and / or computer network. The method can be performed entirely automatically, specifically without user interaction. For example, irradiation and / or image generation can be triggered and / or performed using at least one processor.

[0122] As used herein, the term "trigger" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, one or more of executing and / or performing a specified action, and / or causing execution, initiating execution, or actuating the execution of a specified action. A trigger may include executing at least one piece of software, such as control software, which, when executed by a processing device, causes at least one device or unit to perform specified steps.

[0123] Extracting live data may include extracting material data and / or extracting blood perfusion data, wherein extracting material data includes providing a pattern image to a model and / or receiving material data from the model. Extracting blood perfusion data includes determining the speckle contrast of the pattern image and determining a blood perfusion metric based on the determined speckle contrast. The speckle contrast represents a measure of the average contrast of the intensity distribution within a region of the speckle pattern.

[0124] The method may further include illuminating the user using floodlight from a floodlight source, and simultaneously illuminating the user using the floodlight, generating at least one floodlight image showing at least a portion of the user using an image sensor. The method may further include using the floodlight image to identify the user.

[0125] The illumination time and / or exposure time of the pattern illumination source can depend on the wavelength. For example, for a wavelength of about 1000 nm, 200 to 2000 µs can be used (with or without an OLED). For higher wavelengths, longer exposure times can be used, for example, 7 to 8 times longer.

[0126] The method may include receiving a request to perform at least one operation on the device that requires authentication.

[0127] The method may include: determining whether an object corresponds to a living human based on liveness data; and allowing the user to access one or more functions of the device in response to determining that the user corresponds to a living human.

[0128] A single processing device may be configured to exclusively execute at least one computer program, particularly at least one line of computer program code configured to execute at least one algorithm, as used in at least one embodiment of the method according to the invention. Hereinafter, the computer program executing on the single processing device may include all instructions that cause the computer to perform the method. Alternatively or additionally, at least one method step may be performed using at least one remote device, particularly selected from at least one of a server or a cloud server, especially when the device and the remote device may be part of a computer network. In this case, the computer program may include at least one remote component to be executed by at least one remote processing device to perform at least one method step. Further, the computer program may include at least one interface configured to forward data to and / or receive data from at least one remote component of the computer program.

[0129] This document further discloses and proposes a computer program comprising computer-executable instructions for performing the methods according to the invention in one or more embodiments included herein, when the program is executed on a computer or computer network. Specifically, the computer program may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0130] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" specifically refer to non-transitory data storage devices, such as hardware storage media on which computer-executable instructions are stored. Computer-readable data carriers or storage media can specifically be or may include storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0131] Therefore, specifically, one, more, or even all of the method steps indicated above can be performed by using a computer or computer network, preferably by using a computer program.

[0132] This document further discloses and proposes a computer program product having program code means so that, when the program is executed on a computer or computer network, it performs the method according to the invention in one or more embodiments included herein. Specifically, the program code means may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0133] This document further discloses and proposes a data carrier having a data structure stored thereon, which, after being loaded into a computer or computer network (e.g., into the working memory or main memory of the computer or computer network), can perform methods according to one or more embodiments disclosed herein.

[0134] This document further discloses and proposes a computer program product having program code means stored on a machine-readable medium to perform methods according to one or more embodiments disclosed herein when the program is executed on a computer or computer network. As used herein, a computer program product refers to a program that is a tradable product. The product can generally exist in any format, such as in paper format, or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, the computer program product can be distributed via a data network.

[0135] Finally, this document discloses and proposes a modulated data signal containing computer system or computer network readable instructions for performing methods according to one or more embodiments disclosed herein.

[0136] Referring to the computer implementation aspects of the present invention, one or more, or even all, of the method steps in one or more of the methods disclosed in the embodiments herein can be performed using a computer or computer network. Therefore, typically, any of the method steps involving the provision and / or manipulation of data can be performed using a computer or computer network. Generally, these method steps can include any method steps, except for those that typically require manual work, such as providing samples and / or performing certain aspects of actual measurements.

[0137] Specifically, this article further discloses:

[0138] - A computer or computer network including at least one processor, wherein the processor is adapted to perform a method according to one of the embodiments described in this specification.

[0139] - A computer-loadable data structure adapted to perform a method according to one of the embodiments described in this specification when the data structure is executed on a computer.

[0140] - A computer program, wherein the computer program is adapted, when executed on a computer, to perform a method according to one of the embodiments described in this specification.

[0141] A computer program comprising program means for performing a method according to one of the embodiments described herein when the computer program is executed on a computer or a computer network.

[0142] - A computer program comprising program means according to a preceding embodiment, wherein the program means is stored on a computer-readable storage medium.

[0143] - A storage medium wherein a data structure is stored on the storage medium, and wherein the data structure is adapted to perform a method according to one of the embodiments described herein after being loaded into the main storage device and / or working storage device of a computer or computer network.

[0144] - A computer program product having program code means, wherein the program code means may be stored or stored on a storage medium for performing a method according to one of the embodiments described herein when the program code means is executed on a computer or computer network.

[0145] As used herein, the terms “have,” “include,” or “contain,” or any of their grammatical variations, are used in a non-exclusive manner. Thus, these terms can refer either to a situation where no other features exist in the entity described in the context besides those introduced by these terms, or to a situation where one or more other features exist. For example, the statements “A has B,” “A includes B,” and “A contains B” can refer either to a situation where no other elements exist in A besides B (i.e., A consists solely of B), or to a situation where entity A contains one or more other elements besides B (such as element C, elements C and D, or even other elements).

[0146] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating a feature or element may appear once or more, but are typically used only once when the corresponding feature or element is introduced. In most cases, the expressions "at least one" or "one or more" are not repeated when referring to the corresponding feature or element, but in fact, the corresponding feature or element may appear once or more.

[0147] Furthermore, as used herein, the terms “preferredly,” “more preferably,” “particularly,” “more particularly,” “specifically,” “more specifically,” or similar terms are used in combination with optional features without limiting the possibility of alternatives. Therefore, the features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As those skilled in the art will recognize, the invention can be carried out by using alternative features. Similarly, features introduced by phrases such as “in embodiments of the invention” are intended to be optional features and do not limit any alternative embodiments of the invention, the scope of the invention, or the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0148] In summary, and without excluding other possible embodiments, the following embodiments are conceivable:

[0149] Example 1. An imaging system, comprising

[0150] - At least one pattern illumination source configured to emit at least one light pattern comprising a plurality of light beams having wavelengths in a short-wavelength infrared spectral region, wherein the wavelengths of the light are from 1000 nm to 1200 nm or from 1300 nm to 1500 nm; and

[0151] - At least one image sensor configured to generate at least one pattern image while the pattern illumination source emits the light pattern, wherein the image sensor is at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region.

[0152] Example 2. The imaging system according to the previous embodiment, wherein the wavelength of the light is from 1050 nm to 1150 nm, preferably 1100 nm, or wherein the wavelength of the light is from 1300 nm to 1400 nm.

[0153] Example 3. An imaging system according to any one of the foregoing embodiments, wherein the quantum efficiency of the image sensor is >10%, preferably >20%, and more preferably >40% in the short-wavelength infrared spectral region.

[0154] Example 4. An imaging system according to any one of the foregoing embodiments, wherein the image sensor includes at least one quantum dot sensor.

[0155] Example 5. An imaging system according to any one of the foregoing embodiments, wherein the imaging system is included in a device, wherein the device includes a display, wherein the image sensor is disposed behind the display, wherein the display is at least partially transparent in at least one continuous area covering the image sensor.

[0156] Example 6. The imaging system according to the previous example, wherein the display is or includes at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED) display.

[0157] Example 7. An imaging system according to any one of the foregoing two embodiments, wherein the light pattern passes through the display when illuminated from the pattern illumination source, wherein the display is at least partially transparent in at least one continuous region covering the pattern illumination source.

[0158] Example 8. An imaging system according to any one of the foregoing three embodiments, wherein the imaging system further includes at least one floodlight source configured to emit floodlight.

[0159] Example 9. An imaging system according to the previous embodiment, wherein the floodlight has a wavelength in the short-wavelength infrared spectral region.

[0160] Example 10. An imaging system according to any one of the foregoing two embodiments, wherein the floodlight passes through the display when illuminated from the floodlight source, wherein the display of the device is at least partially transparent in at least one continuous region covering the floodlight source.

[0161] Example 11. An imaging system according to any one of the preceding six embodiments, wherein the device is selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, particularly mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0162] Example 12. An imaging system according to any one of the foregoing embodiments, wherein the pattern illumination source comprises a plurality of light emitters selected from the group consisting of: at least one laser source; at least one laser diode; at least one vertical cavity surface-emitting laser (VCSEL); at least one edge-emitting laser.

[0163] Example 13. An imaging system according to any one of the foregoing embodiments, wherein the imaging system includes at least one optical element configured to modify the light spot generated by the patterned illumination source, wherein the optical element is selected from the group consisting of at least one lens; at least one microlens array (MLA); at least one diffractive optical element (DOE); and at least one metasurface element.

[0164] Example 14. The imaging system according to any one of the foregoing embodiments is used for authenticating users of devices including the imaging system.

[0165] Example 15. A device for authenticating a user of a device to perform at least one operation requiring authentication on the device, the device for authentication comprising:

[0166] - An imaging system according to any one of the foregoing embodiments relating to an imaging system;

[0167] - At least one display, wherein the image sensor is disposed behind the display, wherein the display is at least partially transparent in at least one continuous area covering the image sensor.

[0168] - At least one authentication unit, which is configured to use the pattern image to perform at least one authentication process for a user.

[0169] Example 16. The device according to Example 16 is used for authenticating a user for one or more of the following: in-vehicle payment for vehicle access and / or for starting the vehicle; access control, such as for at least one resource of a mobile device such as a mobile phone; in-cabin sensing; for payment, using a mobile phone and / or a fixed device, such as an ATM, to make a settlement.

[0170] Example 17. A method for a user of an authentication device to perform at least one operation requiring authentication on the device, wherein the method uses the authentication device according to Example 15, and the method includes:

[0171] - Triggers the illumination of the user with a light pattern from the pattern illumination source;

[0172] - While illuminating the user with the light pattern, trigger the generation of at least one pattern image showing at least a portion of the user using the image sensor;

[0173] - Trigger the extraction of liveness data from the pattern image;

[0174] - Use the liveness data to allow or deny the user to perform at least one action on the device that requires authentication.

[0175] Example 18. The method according to the previous embodiment, wherein extracting live data includes extracting material data and / or extracting blood perfusion data, wherein extracting material data includes providing the pattern image to the model and / or receiving material data from the model, wherein extracting blood perfusion data includes determining the speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein the speckle contrast represents a measure of the average contrast of the intensity distribution within the region of the speckle pattern.

[0176] Example 19. The method according to any one of the foregoing embodiments relating to the method, wherein the method further includes illuminating the user with floodlight from the floodlight source, and while illuminating the user with the floodlight, generating at least one floodlight image showing at least a portion of the user with the image sensor, wherein the method further includes using the floodlight image to identify the user.

[0177] Example 20. The method according to any one of the foregoing embodiments relating to the method, wherein the irradiation time and / or exposure time of the pattern irradiation source depends on the wavelength, wherein 200 to 2000 µs is used for a wavelength of about 1000 nm, and wherein a longer exposure time is used for higher wavelengths, for example, 7 to 8 times longer.

[0178] Example 21. The method according to any one of the foregoing embodiments relating to the method, wherein the method includes receiving a request for performing at least one operation requiring authentication on the device.

[0179] Example 22. A method according to any one of the foregoing embodiments relating to the method, wherein the method includes: determining whether the object corresponds to a living human based on the liveness data; and allowing the user to access one or more functions of the device in response to determining that the user corresponds to a living human.

[0180] Example 23. A computer program including instructions that, when executed by a device according to Example 15, cause the device to perform a method according to any one of the foregoing embodiments relating to the method.

[0181] Example 24. A computer-readable storage medium including instructions that, when executed by a device according to Example 15, cause the device to perform a method according to any one of the foregoing embodiments relating to the method.

[0182] Example 25. A non-transient computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any of the embodiments of the foregoing related methods. Attached Figure Description

[0183] Further optional features and embodiments will be disclosed in more detail, preferably in conjunction with the dependent claims, in the following description of embodiments. As those skilled in the art will recognize, the respective optional features can be implemented independently and in any feasible combination. The scope of the invention is not limited to the preferred embodiments. Embodiments are schematically depicted in the accompanying drawings. The same reference numerals in these drawings refer to the same or functionally equivalent elements.

[0184] In the attached diagram:

[0185] Figure 1 An embodiment of a device according to the present invention is shown for authenticating a user of a device to perform at least one operation requiring authentication on the device; and

[0186] Figure 2A flowchart illustrating an embodiment of a method implemented by a user's computer for authenticating a device is shown. Detailed Implementation

[0187] Figure 1 An embodiment of device 110 is illustrated in a highly illustrative manner for authenticating a user to perform at least one operation on the device requiring authentication. The device on which the user wishes to perform the operation may be device 110, or may be included therein, or may be another device. For example, device 110 may be selected from the group consisting of: television devices; game consoles; personal computers; mobile devices, particularly mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0188] The device 110 includes at least one imaging system 111. The imaging system 111 includes at least one pattern illumination source 112, which is configured to emit light having a wavelength in the short-wavelength infrared spectral region. The wavelength of the light is from 1000 nm to 1200 nm or from 1300 nm to 1500 nm.

[0189] Pattern illumination source 112 can be configured to illuminate an object, such as a user, like the user's face, by projecting a light pattern. Illumination can include projecting a light pattern onto the surface of the object. The light pattern can include multiple light spots. The light spots can extend at least partially in space. At least one light spot or any light spot can have an arbitrary shape. In some cases, a circular shape of at least one light spot or any light spot may be preferred. The light pattern can include at least one dot pattern. The light pattern can be a coherent light pattern. The beam of the light pattern can have a single wavelength or multiple wavelengths, for example, to allow for additional measurements in other wavelength channels. The light pattern can include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further embossed patterns. For example, the light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodic 2 / 5 hexagonal pattern can allow for the differentiation of artifacts and available signals. Pattern illumination source 112 can include at least one emitter, and in particular multiple emitters. For example, the transmitter can be selected from the group consisting of: at least one laser source; at least one laser diode; at least one vertical-cavity surface-emitting laser (VCSEL); at least one edge-emitting laser. The transmitter can be used in combination with at least one optical element (such as an MLA, DOE, metasurface, or lens).

[0190] For example, the imaging system 111 may include at least one floodlight source 116 configured to emit floodlight. The floodlight source may include at least one emitter, and more particularly, multiple emitters. The floodlight source may include at least one LED or at least one VCSEL, preferably multiple VCSELs.

[0191] The emission of the floodlight and the illumination of the light pattern can be performed subsequently or at least at a time that overlaps. For example, the floodlight and the light pattern can be emitted simultaneously. For example, one of the floodlight or the light pattern can be emitted at a lower intensity compared to the other.

[0192] Classic 2D deception targets are manufactured to resemble human faces and mimic the reflectivity of human skin in the visible light range. While using NIR (Near-Infrared) light can enhance security, a range of common materials possess the same reflective properties as skin. This invention proposes that the light has wavelengths within the short-wavelength infrared (SWIR) spectral region. The irradiation source 112 can be configured to illuminate the object using a pattern of floodlight and / or a projected light with a center wavelength within the SWIR. Considering SWIR, the above can be altered. Skin is a rather complex composition of different materials, particularly water and lipids. Compared to common materials used for deception targets (e.g., silicon), skin has a different reflectivity value in the SWIR. Therefore, using light within the SWIR allows for better differentiation between human skin and deception targets (e.g., deception targets made of paper, silicon, or resin).

[0193] For example, patterned illumination source 112 can be configured to project a light pattern. The wavelength of the light is from 1000 nm to 1200 nm. For example, the wavelength of the light is from 1050 nm to 1150 nm, preferably 1100 nm. In these wavelength regions, the reflective behavior of skin differs from that of other materials. For example, the wavelength of the light is from 1300 nm to 1500 nm, preferably from 1300 nm to 1400 nm. In these wavelength regions, skin has distinctly different reflectivity values.

[0194] Furthermore, the spot structure of the light spot on the surface of the object is suitable for material detection in the wavelength region, especially using beam profiling analysis.

[0195] For example, floodlight source 116 can be configured to illuminate an object using floodlight. The center wavelength of the floodlight and the center wavelength of the light pattern can be the same, for example, within ±100 nm. However, other embodiments are conceivable. For example, the wavelength of the light is from 1300 nm to 1500 nm, preferably from 1300 nm to 1400 nm. For this wavelength range, skin has clearly distinguishable reflectivity values. In particular, due to the presence of water and lipids, skin reflects light differently from artificial materials, especially less than artificial materials.

[0196] Therefore, this invention proposes using short-wave infrared light to illuminate an object presented to a camera to verify that the object is a living organism and not a deceptive object, such as a mask. Using these wavelengths is advantageous because light in these wavelength ranges is invisible to humans, and this wavelength can be associated with low intensity in the solar spectrum. Therefore, using the aforementioned wavelength range can provide the advantage of resisting solar radiation on the object.

[0197] The imaging system 111 further includes at least one image sensor 118, which is configured to generate at least one image while the patterned illumination source 112 emits light. The image sensor 118 is at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region.

[0198] As mentioned above, higher wavelengths offer several advantages. However, detectors typically used in authentication systems quickly become inefficient in these wavelength ranges, making the use of higher wavelengths problematic. Image sensor 118 may include at least one quantum dot sensor. For example, the quantum dot sensor may be designed as described in US 2023 / 258498A1. Image sensor may include a substrate made of, for example, a semiconductor material (e.g., silicon). The substrate may be made of an insulating layer (e.g., made of silicon oxide) and conductive traces and vias (e.g., made of copper). The substrate may be covered by the insulating layer. The insulating layer may be made of silicon oxide. Conductive vias may extend through the insulating layer. The vias may be made of copper or tungsten. Image sensor 118 may include at least one via for each pixel. The location of each pixel may position one via. The via for each pixel may correspond to an electrode of the pixel. Image sensor 118 may further include a layer covering the insulating layer and all vias corresponding to the pixels. The layer may be continuous at all pixel locations. The layer may contact the vias. The layer may be made of quantum dots. Quantum dots can be fixed together with a resin or matrix and attached to an insulating layer. Preferably, the quantum dot layer may contain only quantum dots and resin. All quantum dots in the layer may be substantially identical. All quantum dots in the quantum dot layer have substantially the same size and are made of the same composition. For example, all quantum dots in the quantum dot layer are composed of lead sulfide (PbS). Quantum dots, or semiconductor nanoparticles, are nanomaterial structures that generate electron-hole pairs when a given photon is incident on the nanomaterial structure. In this way, for example, detection elements, such as photodetectors, can be generated based on semiconductor nanoparticles. Quantum dots include a semiconductor core. Quantum dots may also include a shell surrounding the core to protect and passivate it, preferably made of a semiconductor material. Quantum dots further include ligands, organolithic compounds, organometallic compounds, or inorganic molecules extending from the shell and passivating, protecting, and functionalizing the semiconductor surface. The composition of quantum dots may be selected from the following materials. The core is made of, for example, materials or alloys of the following: CdSe, CdS, CdTe, CdSeS, CdTeSe, AgS, ZnO, ZnS, ZnSe, CuInS, CuInSe, CuInGaS, CuInGaSe, PbS, PbSe, PbSeS, PbTe, InAsSb, InAs, InSb, InGaAs, InP, InGaP, InAlP, InGaAlP, InZnS, InZnSe, InZnSeS, HgTe, HgSe, HgSeTe, Ge, Si.The shell is made of, for example, materials or alloys of the following: CdSe, CdS, CdTe, CdSeS, CdTeSe, AgS, ZnO, ZnS, ZnSe, CuInS, CuInSe, CuInGaS, CuInGaSe, PbS, PbSe, PbSeS, PbTe, InAsSb, InAs, InSb, InGaAs, InP, InGaP, InAlP, InGaAlP, InZnS, InZnSe, InZnSeS, HgTe, HgSe, HgSeTe, Ge, Si.

[0199] The use of quantum dot image sensors allows for efficient image detection even in SWIR. The quantum efficiency of the image sensor can be >10%, preferably >20%, and more preferably >40% in the short-wavelength infrared spectral region.

[0200] Device 110 further includes at least one display 120, wherein an image sensor is disposed behind display 120. Display 120 can be configured to display information. This information can be any type of information, such as at least one image, at least one chart, at least one histogram, at least one graph, text, numbers, at least one symbol, operation menus, etc. Display 120 can be or may include at least one display panel. Display 120 can have any shape, such as a rectangular shape. The display can be a front-facing display of the device. Display 120 can include at least one of a display panel (particularly including multiple pixels and / or multiple transistors) or glass (specifically, a cover glass, particularly configured to cover the display panel). Display 120, specifically the display panel, can be or may include at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED).

[0201] Pattern illumination source 112 and / or image sensor 118 may be arranged behind display 120. Light may pass through display 120 when illuminated from pattern illumination source 112. Display 120 may be at least partially transparent. Display 120 may be at least partially transparent over at least one continuous area covering pattern illumination source 112, flood illumination source 116, and / or image sensor 118. For example, display 120 may include perforations covering a continuous area of ​​pattern illumination source 112, flood illumination source 116, and / or image sensor 118. For example, display 120 may have a transmittance of less than or equal to 20%, preferably less than or equal to 15%, more preferably less than or equal to 10%. For example, the intensity of the light beam after being projected through display 120 may correspond to ≤ 10% of the intensity associated with the light beam at the time of emission.

[0202] Device 110 further includes at least one authentication unit 122, which is configured to perform at least one authentication process 124, particularly a computer-implemented method for authenticating users according to the present invention, such as... Figure 2 As shown in the embodiment. The authentication unit 122 may include at least one processor.

[0203] These method steps can be executed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be executed repeatedly.

[0204] The method includes:

[0205] - (128) Trigger the illumination of the user with a light pattern from the pattern illumination source 112;

[0206] - (130) While illuminating the user with the light pattern, trigger the generation of at least one pattern image showing at least a portion of the user using the image sensor;

[0207] - (132) Trigger the extraction of liveness data from the pattern image;

[0208] - (126) Use the liveness data to allow or deny the user to perform at least one operation on the device that requires authentication.

[0209] The authentication unit 122 can be configured to extract liveness data from a pattern image. The authentication unit 122 can be configured to use the liveness data to allow or deny a user from performing at least one operation on the device that requires authentication. Specifically, extracting liveness data includes extracting material data and / or extracting blood perfusion data.

[0210] Material data can include information about the material type of an object's surface under illumination, for example, by a light pattern. Extracting material data can be or can include generating data about the material type and / or deriving data from the material type. Material data can include information about the material type of the object. Advantageously, material detection must only work on targets with reflectivity similar to human skin.

[0211] Material data can be extracted from a pattern image. The material data can indicate the type of material. In particular, the material data can indicate whether an object associated with the image at least partially comprises skin. The material data can be associated with an object, especially with an object shown in the image. Extracting material data from a pattern image can include generating material type and / or data derived from the material type.

[0212] Material data extraction can be based on pattern images, more preferably on one or more partial images. For example, the image can be reduced to a predefined size by applying one or more image processing techniques. Reduction may include selecting at least one region of interest and segmenting the pattern image into regions of the predefined size pattern image. The regions of the predefined size image may be associated with an object. Image processing techniques may include at least one image recognition technique, particularly facial recognition technique. Image recognition techniques include at least one process for identifying a user in an image. Image recognition may include at least one technique using techniques selected from the following: color-based image recognition, such as using features like template matching; segmentation and / or connected component (blob) analysis, such as using size or shape; machine learning and / or deep learning, such as using at least one convolutional neural network. Parts of the image other than the regions of the predefined size image may be associated with the background and / or may be unrelated to the object. Image portions useful for subsequent analysis can be selected.

[0213] Material data may include at least one measure of reflectance. At least one algorithm can be used to evaluate the material data (e.g., reflectance), wherein the algorithm is designed to check whether the object is a deceptive target. Alternatively or alternatively, artificial neural networks can be used to check whether an image is a deceptive target. Refer to the description above for information on material data extraction.

[0214] Determining whether a surface is human skin may include comparing extracted material data with skin-related material data (skin material data). Comparing the material data with desired material data may include determining the similarity between the extracted material data and the skin material data. Skin material data may refer to predetermined material data of skin. For example, material data may include reflectance values, and the method may include comparing the determined reflectance values ​​of the surface with at least one range of reflectance values ​​of skin. The method may include considering tolerances such as ±10%, preferably ±5%, more preferably ±1%.

[0215] Skin material data may be stored in at least one database and / or may be retrieved from at least one database, for example, the database may be at least partially cloud-based, for example via at least one communication interface 134.

[0216] If the material data matches the skin's material data, the object is determined to be a human. Otherwise, if the material data does not match the skin's material data, the object is determined to be a non-living organism. The comparison of the material data with the skin's material data can lead to allowing and / or denying the user and / or object from performing at least one operation requiring authentication, as described in more detail below.

[0217] For example, extracting material data from patterned images can include beam profile analysis of the light spot. For more information on beam profile analysis, see WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference.

[0218] This method may include determining the distance between an object and an image sensor 118. Reflectivity may be calculated considering the distance. The distance may be considered a corrected value for a reflectivity metric. This distance may be used to determine whether the distance between the object and the image sensor 118 is within the working range of a model used to extract material data. Distance is also important for facial authentication because the facial recognition model is associated with a working range. This working range may specify a range of distances between the object and the image sensor, where the model operates and / or is trained on the user's image. In embodiments, the working range may specify at least one upper boundary and / or at least one lower boundary of the distance between the object and the image sensor and / or the illumination source. The working range may be associated with an authentication process. The working range may include at least one value. This value may be numerical, particularly a positive value. An indication of the working range may be received, particularly before determining whether the distance is within or outside the working range of the authentication process. The indication of the working range may be adapted to determine whether the distance is within or outside the working range of the authentication process. The indication of the working range may be adapted to compare the distance with the working range. At least one distance determination technique may be used to calculate the distance. For example, distance can be determined using one or more of the following: beam profiling analysis (e.g., as described in WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO2018 / 091640 A1, the entire contents of which are incorporated herein by reference), time of flight, triangulation, etc. This distance can be calculated using a distance sensing system of a mobile device, which includes an illumination source and an image sensor.

[0219] The operation requiring authentication on the device can be any operation that requires access to at least one resource associated with the device. The method may include receiving a request for access to at least one resource associated with the device and performing at least one authentication process. A communication interface 134 (e.g., a user interface) may be configured to receive requests for access to at least one resource associated with the device, particularly for performing at least one operation requiring authentication on the device.

[0220] For example, the authentication process may include performing at least one face detection step. The face detection step may include analyzing at least one image of the user, generated, for example, by an image sensor or another camera. The image may be a floodlight image. Determining whether a user corresponds to an authorized user can be performed using an image of the user generated simultaneously with the object being illuminated by SWIR. This allows the imaging to be undetectable to humans and, to a very small extent, included within the solar spectrum. The authentication process may include generating at least one floodlight image showing the associated user simultaneously with the user being illuminated by floodlight, and determining, based on the floodlight image, whether the user's identity corresponds to a verified identity. The authentication process may include allowing the user access to the resource if the user's identity corresponds to a verified identity, otherwise denying the user access to the resource if the user's identity does not correspond to a verified identity. The method may include:

[0221] -Use floodlight illumination on the user by using floodlight source 116;

[0222] - Capture at least one floodlight image by using image sensor 118 or another camera of device 110.

[0223] The face detection step may include analyzing a floodlight image. For example, an authentication process may include performing at least one face detection using a floodlight image. Face detection can be performed locally on the device. However, face recognition (i.e., assigning identity to detected faces) can be performed remotely, for example, in the cloud, especially when identification and not just verification are required. User templates can be stored at a remote device, for example, in the cloud, and do not need to be stored locally. This can be advantageous from a storage and security perspective. The authentication process may include identifying a user based on a floodlight image. Identification may include assigning identity to detected faces and / or verifying the user's identity. Identification may include performing face verification on the imaged face to confirm whether it is the user's face. Identifying a user may include matching a floodlight image (e.g., showing the outline of parts of the user, particularly parts of the user's face) with a template. To match the floodlight image with the template, the similarity between at least one image feature vector obtained from the floodlight image and at least one template feature vector may be considered and / or evaluated. The template vector may be obtained from a template image. The template image may be generated during the registration process. Face detection may include analyzing a floodlight image. Specifically, the analysis of floodlight images may include the use of at least one image recognition technique, particularly facial recognition technology. Image recognition technology includes at least one process for identifying a user in an image. Image recognition may include at least one technique using a combination of the following: color-based image recognition, for example, using features such as template matching; segmentation and / or connected component (blob) analysis, for example, using size or shape; machine learning and / or deep learning, for example, using at least one convolutional neural network.

[0224] For example, authentication may include identifying a user. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing facial verification on the imaged face to confirm whether it is the user's face. Identifying a user may include matching a flood image (e.g., showing the outlines of various parts of the user, particularly the outlines of various parts of the user's face) with a template (e.g., a template image generated during registration). Identifying a user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the flood image can match the image template, authentication may be successful. If the flood image cannot match the image template, authentication may be unsuccessful.

[0225] Performing the method according to the invention allows for the determination of whether an object corresponds to a living organism. The authentication process may include allowing the user access to the resource if it is determined that the user's pattern image corresponds to a living organism, otherwise denying the user access to the resource if it is determined that the user's pattern image does not correspond to a living organism. For example, the authentication process may be verified based on extracted material data. Verification may include determining the similarity between the extracted material data and skin, for example, comparing the extracted material data with skin material data. The comparison of material data with skin may result in allowing and / or denying the user and / or object from performing at least one operation requiring authentication. In an example, skin may be compared as desired material data with a non-skin material or silicon as material data, and the result may be rejection because silicon or non-skin materials may differ from skin.

[0226] The authentication unit can be further configured to determine a depth map. The determined depth map can be compared to a predetermined depth map of the user, for example, determined during the registration process. The authentication unit can be configured to authenticate the user if the determined depth map matches (in particular, at least within tolerance) the user's predetermined depth map. Otherwise, the user can be rejected.

[0227] Allowing a user to access resources may include authorizing the user. Device 110 may include at least one authorization unit 136 configured to allow the user to perform at least one operation on the device, such as unlocking the device, upon successful authentication, or to deny the user from performing at least one operation on the device if authentication fails. Thus, the user is aware of the authentication result. Authorization unit 136 may be configured to allow or deny the user to perform at least one authentication-required operation on the device based on extracted material data and identification, such as using floodlight images. Authorization unit 136 may be configured to allow or deny the user access to one or more functions associated with the device, depending on authentication or denial. Allowing may include granting permission to access the one or more functions. Authorization unit 136 may be configured to determine whether the user corresponds to an authorized user, wherein allowing or denying is further based on determining whether the user corresponds to an authorized user. Authorization unit 136 may include at least one processor or may be designed as software or an application. Authorization unit 136 and authentication unit 122 may be integrated, for example, by using the same processor.

[0228] List of reference numerals

[0229] 110 Devices used for authenticating users

[0230] 111 Imaging System

[0231] 112 pattern illumination source

[0232] 116 floodlight source

[0233] 118 Image Sensor

[0234] 120 monitor

[0235] 122 Authentication Unit

[0236] 124 Authentication Process

[0237] 126 Allow or deny a user to perform at least one action on the device

[0238] 128 trigger irradiation

[0239] 130 triggers the generation of at least one image

[0240] 132 triggers the extraction of live data

[0241] 134 communication interface

[0242] 136 authorized units.

Claims

1. An imaging system (111), comprising: - At least one pattern illumination source (112), the at least one pattern illumination source being configured to emit at least one light pattern, the at least one light pattern comprising a plurality of light beams having wavelengths in the short-wavelength infrared spectral region, wherein, The wavelength of this light is from 1000 nm to 1200 nm or from 1300 nm to 1500 nm; and - At least one image sensor (118) configured to generate at least one pattern image while the pattern illumination source (112) emits the light pattern, wherein the image sensor (118) is at least partially sensitive to electromagnetic radiation in the short-wavelength infrared spectral region.

2. The imaging system (111) according to the preceding claim, wherein, The wavelength of the light is from 1050 nm to 1150 nm, preferably 1100 nm, or wherein the wavelength of the light is from 1300 nm to 1400 nm.

3. The imaging system (111) according to any one of the preceding claims, wherein, The quantum efficiency of the image sensor (118) is >10%, preferably >20%, and more preferably >40% in the short-wavelength infrared spectral region.

4. The imaging system (111) according to any one of the preceding claims, wherein, The image sensor (118) includes at least one quantum dot sensor.

5. The imaging system (111) according to any one of the preceding claims, wherein, The imaging system (111) further includes at least one floodlight source (116) configured to emit floodlight, wherein the floodlight has a wavelength in the short-wavelength infrared spectral region.

6. The imaging system (111) according to any one of the preceding claims is used for authenticating the user of the device, the device including the imaging system.

7. A device (110) for authenticating a user of a device to perform at least one operation requiring authentication on the device, the device (110) for authentication comprising: -The imaging system (111) according to any one of the preceding claims relating to the imaging system (111). - At least one display (120), wherein the image sensor (118) is arranged behind the display (120), wherein the display (120) is at least partially transparent in at least one continuous area covering the image sensor (118). - At least one authentication unit (122) is configured to use the pattern image to perform at least one authentication process for a user.

8. The device (110) of claim 7 is used for authenticating a user for one or more of the following: in-vehicle payment for vehicle access and / or for starting the vehicle; access control, such as for at least one resource of a mobile device such as a mobile phone; in-cabin sensing; for payment, using a mobile phone and / or a fixed device, such as an ATM, to make a settlement.

9. A method for a user of an authentication device to perform at least one operation requiring authentication on the device, wherein, The method uses the device (110) for authentication as described in claim 7, and the method includes: - (128) Trigger the illumination of the user with a light pattern from the pattern illumination source; - (130) While illuminating the user with the light pattern, trigger the generation of at least one pattern image showing at least a portion of the user using the image sensor; - (132) Trigger the extraction of liveness data from the pattern image; - (128) Use the liveness data to allow or deny the user to perform at least one operation on the device that requires authentication.

10. The method according to the preceding claim, wherein, Extracting live data includes extracting material data and / or extracting blood perfusion data, wherein extracting material data includes providing the pattern image to the model and / or receiving material data from the model, and extracting blood perfusion data includes determining the speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein the speckle contrast represents a measure of the average contrast of the intensity distribution within the region of the speckle pattern.

11. The method according to the preceding claim, wherein, The material data includes information about the material of the irradiated user, and / or includes information about the type of material of the user's surface under irradiation.

12. The method according to any one of the preceding claims relating to the method, wherein, The method further includes illuminating the user with floodlight from the floodlight source, and while illuminating the user with the floodlight, generating at least one floodlight image showing at least a portion of the user with the image sensor, wherein the method further includes using the floodlight image to identify the user.

13. A computer program comprising instructions that, when executed by the device (110) according to claim 7, cause the device (110) to perform the method according to any one of the preceding claims relating to the method.

14. A computer-readable storage medium comprising instructions that, when executed by the device (110) according to claim 7, cause the device (110) to perform the method according to any one of the preceding claims relating to the method.

15. A non-transient computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of the preceding claims relating to the method.

Images