Facial gesture recognition in SWIR images

Abstract

To provide a system and method for analyzing an image to recognize facial expression.SOLUTION: A system 100 provided herein comprises: at least one short wave infrared (SWIR) illumination element configured to generate SWIR illumination for illuminating a face of a person, at least one SWIR sensor configured to capture at least one SWIR image of the face under the SWIR illumination, and a non-transitory medium storing program instructions designed to cause a processor to analyze the at least one SWIR image to recognize facial expression depicted by the face when executed by a processor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention, in some embodiments thereof, relates to image processing and more particularly, but not exclusively, to automatic facial expression recognition of faces depicted in images. [Background technology]

[0002] Related Applications This application claims the benefit of priority to U.S. Patent Application No. 17 / 689,109, filed March 8, 2022, the entire contents of which are incorporated herein by reference.

[0003] Understanding facial expressions and gestures is very important in human communication. Attempts have been made to automatically recognize facial expressions depicted in images. Summary of the Invention

[0004] According to a first aspect, a system for analyzing images to recognize facial expressions includes at least one short-wave infrared (SWIR) illumination element that generates SWIR illumination to illuminate a human face, at least one SWIR sensor that captures at least one SWIR image of the face under the SWIR illumination, and a non-transitory medium storing program instructions that, when executed by a processor, cause the processor to analyze the at least one SWIR image to recognize facial expressions depicted by the face.

[0005] In a further embodiment of the first aspect, the apparatus further comprises at least one filter that filters out electromagnetic radiation of wavelengths that are not significantly absorbed by water vapor in the air depicted in the scene.

[0006] In a further embodiment of the first aspect, the at least one filter comprises a spectral narrow passband filter that is one of: (i) a filter that passes wavelengths between about 1350 and 1450 nanometers (nm) and excludes wavelengths above about 1450 nm and below about 1350 nm; and (ii) a filter that passes wavelengths between about 1360 nm and 1380 nm and excludes wavelengths above about 1380 nm and below about 1360 nm.

[0007] A further implementation form of the first aspect further includes controlling the SWIR lighting elements to generate a target lighting pattern for illuminating the face, and capturing at least one SWIR image while the target lighting pattern is being generated.

[0008] In a further embodiment of the first aspect, the target illumination pattern is selected from the group comprising pulsating, continuous, polarized, diffuse, and time modulated.

[0009] In a further embodiment of the first aspect, the SWIR lighting element generates SWIR illumination at an intensity that does not elicit a facial expression indicating discomfort, and a visible light lighting element that generates visible light illumination at the intensity of the SWIR lighting element elicits a facial expression indicating discomfort.

[0010] In a further embodiment of the first aspect, the system further comprises a visible light image sensor for capturing at least one visible light image, and the program instructions are for analyzing a combination of the at least one SWIR image and the at least one visible light image.

[0011] In a further embodiment of the first aspect, the at least one SWIR lighting element generates illumination in a band of wavelengths invisible to the human eye, and the at least one SWIR sensor senses in a band of wavelengths invisible to the human eye.

[0012] In a further embodiment of the first aspect, the band of wavelengths comprises about 1000 to 2000 nm.

[0013] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to distinguish facial hair on the face from surrounding skin.

[0014] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to distinguish makeup on the face from surrounding skin.

[0015] In a further embodiment of the first aspect, the at least one SWIR image includes a plurality of SWIR frames of a SWIR video depicting an initial state of the face, a transition from the initial state to a facial expression, and a face depicting the facial expression, and the program instructions include instructions for analyzing the plurality of SWIR frames to detect the transition from the initial state to the facial expression.

[0016] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to distinguish facial regions having high moisture content from adjacent facial regions having low moisture content.

[0017] In a further embodiment of the first aspect, the area of ​​the face having high moisture content comprises the lips and the area of ​​the face having low moisture content comprises the skin surrounding the lips.

[0018] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to distinguish ears from adjacent hair.

[0019] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to detect at least one of a tip of the nose, a chin, and a philtrum.

[0020] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to distinguish teeth from surrounding lips.

[0021] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to detect facial wrinkle lines under the eyes and / or near the nose.

[0022] In a further embodiment of the first aspect, the program instructions further include instructions for analyzing the at least one SWIR image to detect areas of skin that are darker than surrounding adjacent skin.

[0023] According to a second aspect, a method for training a machine learning (ML) model for recognizing facial expressions includes creating a training dataset of multiple records, wherein records of the training dataset of multiple records include a plurality of facial expression features extracted from at least one SWIR image of a face of a sample individual under SWIR illumination and ground truth labels indicative of facial expressions depicted by the sample individual; and training the machine learning model with the training dataset of multiple records to generate target facial expression results for a target individual in response to input of a plurality of target facial expressions extracted from at least one SWIR image of a target face of the target individual under SWIR illumination.

[0024] According to a third aspect, a method of analyzing an image to recognize facial expressions includes providing a plurality of target facial expression features extracted from at least one SWIR image of a target face of a target individual under SWIR illumination to a machine learning model trained according to the second aspect, and obtaining a target facial expression of the target individual as a result of the machine learning model.

[0025] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the present patent specification, including definitions, will control. Furthermore, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0026] Some embodiments of the present invention are herein described, by way of example only, with reference to the accompanying drawings. Referring now in detail to the drawings, it is emphasized that the particulars shown are exemplary and are for illustrative purposes only of embodiments of the invention. In this regard, the description with reference to the drawings will make apparent to those skilled in the art how embodiments of the invention may be practiced. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram of a system for analyzing SWIR images to recognize facial expressions, according to some embodiments of the present invention. [Figure 2A] 1 is a flowchart of a method for analyzing SWIR images to recognize facial expressions, according to some embodiments of the present invention. [Figure 2B] 1 is a flowchart of a method for training a machine learning model for facial expression recognition using SWIR images, according to some embodiments of the present invention. [Figure 2C] 1 is a flowchart of a method for analyzing SWIR images to recognize facial expressions according to some embodiments of the present invention. [Figure 3] 1 is a SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 4] 10 is another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 5] 10 is yet another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 6] 10 is yet another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 7] 10 is yet another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 8] 10 is yet another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 9] 10 is yet another SWIR image of a person's face under SWIR illumination and a visible light image of the same person, according to some embodiments of the present invention. [Figure 10] 1 is a sequence of SWIR images of a person's face under SWIR illumination taken while smiling, and a sequence of visible light images of the same person, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] The present invention, in some embodiments thereof, relates to image processing and more particularly, but not exclusively, to automatic facial expression recognition of faces depicted in images.

[0029] As used herein, the terms "facial expression," "emotional expression," and "facial gesture" are used interchangeably. Images described herein may be analyzed to determine facial expressions and / or emotional expressions, such as happiness, sadness, anger, confusion, pain, surprise, fear, disgust, neutral, etc., and / or facial gestures.

[0030] As used herein, the term SWIR is an exemplary embodiment of wavelengths that are invisible to the human eye. Note that other wavelengths that are not SWIR and are invisible to the human eye can be used.

[0031] Some embodiments of the present invention relate to systems, methods, devices, and / or code instructions (stored on a data storage device and executable by one or more hardware processors) for automatically analyzing short-wavelength infrared (SWIR) images to recognize facial expressions. One or more SWIR illumination elements generate SWIR illumination for illuminating a person's face. Because SWIR illumination is invisible and / or harmless to the eyes, the SWIR illumination can be more intense and / or directed toward the face. One or more SWIR sensors capture one or more SWIR images of the face under SWIR illumination. The SWIR-illuminated images may be well illuminated under SWIR illumination compared to visible light images where similar illumination intensity and / or orientation cannot be achieved, for example, due to discomfort to the person and / or risk of eye damage. When brighter and / or more direct SWIR illumination is applied, a shorter camera exposure time may be selected and / or a longer camera range may be obtained, and / or finer details may be depicted in the captured SWIR images. The SWIR image(s) are automatically analyzed to recognize facial expressions displayed by a person's face, optionally based on an analysis of facial features extracted from the SWIR image. The SWIR image can increase the accuracy of facial expressions detected by the automatic facial expression recognition process. Higher accuracy can be achieved, for example, based on extracted facial features that are well depicted in the SWIR image compared to visible light images, which may not be well illuminated and therefore may not be able to extract corresponding facial features, or extract them as reliably. In some embodiments, facial features that can be extracted from the SWIR image do not have corresponding features extractable from visible light-illuminated images, since SWIR-extractable features are generated from effects of SWIR illumination that are not present in the visible light image. For example, features described herein such as reflection from the tip of the nose or strong brightness of tissue with high water content compared to tissue with low water content. In some embodiments, a combination of features extracted from the SWIR image and the visible light image is analyzed.

[0032] At least some embodiments described herein address the technical challenge of improving computing devices that analyze images to automatically recognize facial expressions. At least some embodiments described herein improve the technical field of automatic image processing for automatic facial expression recognition. Improvements can be, for example, improved accuracy of automatic recognition, reduced training dataset size for training machine learning models for automatic recognition, and / or extraction of facial features for automatic recognition that improve recognition accuracy. Using a standard approach, a visible light sensor (e.g., a standard camera) is used to capture visible light images of an individual's face. The facial images are analyzed to automatically recognize facial expressions. Visible light has several disadvantages. For example, visible light cannot be directed at an individual's eyes and / or cannot be set to a high intensity. Such intense light and / or light directed at an individual can cause discomfort to the individual, causing their facial expression to change from a previous state (that would have been analyzed) to a current state indicating discomfort. Furthermore, the use of intense light is limited by potential damage to the eyes. This limits the use of external lighting to improve visibility without discomfort. Furthermore, some facial features are technically difficult to identify, extract, and / or segment under visible light. For example, facial hair, makeup, and ears may be misidentified, e.g., confused with other nearby tissue. In another example, some facial features, such as facial lines and / or wrinkles, the tip of the nose, the chin, the philtrum, and tissues with high water content (e.g., the eyeballs, lips, and nasal openings), may not be sufficiently visible under visible light to allow detection and / or extraction. In another example, visible light sources, whether natural (e.g., the sun, fire) or artificial (e.g., light bulbs, LEDs), can create uneven lighting and / or shadows, which can be technically challenging for automated feature detection and / or extraction. For example, shadows and / or uneven lighting may be misidentified as facial expressions when they are not clearly so.

[0033] In at least some embodiments, improvements are obtained by analyzing SWIR images of a face under SWIR illumination. One or more SWIR illumination elements can be controlled to generate target SWIR illumination designed to delineate facial features in the captured SWIR image. Facial features under SWIR illumination may be more distinct from other nearby facial features, which may enable improved feature extraction and / or segmentation. For example, improving the ability to extract and / or segment one or more of the following: facial hair vs. nearby skin, makeup, ears vs. hair, nose tip, chin, facial lines and / or wrinkles, philtrum, and tissues with high water content (e.g., eyes, lips, nasal openings). SWIR illumination is invisible and / or safe to the human eye. This allows the SWIR illumination to be controlled and / or configured in various target illumination patterns without eliciting other facial expressions of the subject being illuminated for image capture. For example, SWIR illumination can be positioned and / or directed directly in front of an individual's face and / or set to a high intensity without the individual perceiving the SWIR illumination. Because the individual does not perceive the SWIR illumination, the individual does not feel uncomfortable, allowing the individual to show their true facial emotions rather than emotions corresponding to the discomfort of external illumination sources. Furthermore, SWIR illumination can be set to provide even illumination and / or avoid shadows, as described herein, thereby reducing or avoiding technical challenges in facial feature extraction and / or facial feature detection. Because facial features under SWIR illumination appear more uniform across a variety of subjects compared to facial features under visible light illumination, fewer facial features are required under SWIR illumination (e.g., to account for variations in various visible light illumination scenarios) compared to visible light illumination, thereby reducing the size of training datasets for training machine learning models for automatic facial expression recognition.

[0034] Before describing at least one embodiment of the present invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangements of components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0035] The present invention may be a system, a method, and / or a computer program product, which may include computer-readable storage medium(s) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0036] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific, non-exhaustive examples of computer-readable storage media include: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0037] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0038] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to implement aspects of the present invention.

[0039] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0040] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, which having instructions stored therein, can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, including an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0041] Furthermore, computer-readable program instructions can be loaded into a computer, other programmable data processing apparatus, or other device to execute a series of operational steps on the computer, other programmable data processing apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0042] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or a combination of special-purpose hardware and computer instructions.

[0043] Reference is now made to FIG. 1 , which is a block diagram of a system for analyzing SWIR images to recognize facial expressions, according to some embodiments of the present invention. Facial expression systems using visible light that rely on additional artificial lighting to illuminate the face are limited in that the artificial lighting can be uncomfortable and / or damaging to the human eye. Powerful light sources and / or light sources positioned toward the human face, which are required to generate good images for automatic facial expression recognition, cannot be used. Components of the present system described with reference to FIG. 1 and / or features of the present method described with reference to FIGS. 2A-2C and / or 3-10 use SWIR illumination. Because SWIR illumination is invisible and / or not harmful to the eye, it can be highly intense and / or directed toward the face. SWIR illumination makes it possible to generate well-illuminated SWIR images that can be used for automatic facial expression recognition.

[0044] As used herein, the term "solar blind region" refers to a wavelength spectrum where electromagnetic radiation (e.g., generated by sunlight and / or artificial light sources) is largely (e.g., mostly) absorbed by the atmosphere (e.g., by water vapor in the air) and / or where there is little emission of electromagnetic radiation, for example, a range of about 1350-1450 nm (optionally 1360-1380 nm).

[0045] Further details of solar blind areas are described with reference to U.S. Patent Application No. 17 / 689,109, entitled "SOLAR BLIND IMAGING," filed March 8, 2022, the entire contents of which are incorporated herein by reference.

[0046] The system 100 includes one or more SWIR sensor(s) 104 that capture SWIR image(s) of a scene in the SWIR wavelength range (optionally about 1000-2000 nanometers (nm)). The SWIR wavelength range may include solar blind regions. Examples of the SWIR sensor(s) 104 include plasmon-based CMOR, barometer array-based FIR, and 3D passive imaging.

[0047] It should be noted that other image sensors can be used that capture wavelengths that are not visible to the human eye, and / or other illumination elements can be used that generate illumination at wavelengths that are not visible to the human eye and that are sensed by the image sensor to generate an image.

[0048] For clarity and simplicity of explanation, as used herein, the term SWIR sensor refers to one or more SWIR sensors, and the term SWIR illumination element refers to one or more SWIR illumination elements.

[0049] System 100 can include one or more SWIR illumination element(s) 106 that generate SWIR illumination in a SWIR wavelength range (optionally about 1000-2000 nm) detectable by SWIR sensor 104. SWIR illumination element(s) 106 can generate at least some SWIR illumination in the solar blind region, i.e., a region having wavelengths in the range of about 1350-1450 nm, or about 1360 nm-1380 nm.

[0050] The SWIR illumination element 106 may include an artificially controlled illumination source with emission in the non-visible range designed to generate a targeted illumination pattern, e.g., pulsed, continuous, polarized, diffuse, and / or strobe. Because the SWIR illumination generated by the SWIR illumination element 106 is not detectable by the human eye and / or other visible wavelength sensors, the brightness of the SWIR illumination element 106 can be much higher than, for example, a visible light source. SWIR illumination in a wavelength range invisible to the human eye may be much more powerful than light sources in other wavelengths, such as the visible light range, while still maintaining eye protection within safety limits and / or avoiding subject discomfort. When brighter and / or more direct SWIR illumination is applied, a shorter camera exposure time may be selected and / or a longer camera range may be obtained, and / or finer details may be depicted in the captured SWIR image.

[0051] The system 100 can include one or more filters 108 that filter out electromagnetic radiation at wavelengths that are visible to the human eye, e.g., wavelengths that are not significantly absorbed by water vapor in the air depicted in the scene. The filters 108 can pass electromagnetic radiation at wavelengths that are not significantly visible to the human eye, e.g., that are absorbed by water vapor in the air depicted in the scene.

[0052] Filter 108 is designed to filter out wavelengths, for example in the visible light range, that may cause glare and / or other artifacts when reflected from the scene.

[0053] The filter 108 may include a spectrally narrow passband filter that passes wavelengths in the SWIR wavelength range of approximately 1000-2000 nm. The filter 108 may pass wavelengths in the solar blind region, for example, approximately 1350-1450 nanometers, or approximately 1360-1380 nm. The filter may exclude wavelengths above the SWIR wavelength range of approximately 2000 nm, or above the solar blind region of approximately 1450 nm, or approximately 1380 nm, or other values. The filter may exclude wavelengths below the SWIR wavelength range of approximately 1000 nm, or below the solar blind wavelength range of approximately 1360 nm, or 1350 nm, or other values.

[0054] It should be noted that in some embodiments, filter 108 may be omitted and / or may be incorporated into SWIR sensor 104 and / or SWIR lighting element 106. For example, SWIR sensor(s) 104 are designed to capture electromagnetic energy having wavelengths in the solar blind region without necessarily requiring a filter and / or using a built-in filter. In another example, SWIR lighting element 106 is designed to generate electromagnetic energy having wavelengths in the solar blind region without necessarily requiring a filter and / or using a built-in filter.

[0055] The SWIR wavelength range emitted by the SWIR illumination element 106 includes at least the wavelength band that passes through the filter 108 and is sensed by the SWIR sensor 104 .

[0056] The SWIR illumination generated by the SWIR illumination element 106 passes through a filter 108 at some point before reaching the SWIR sensor 104, on its path away from the SWIR illumination element 106 towards objects in the scene (i.e., before being reflected by objects in the scene) and / or on its path towards the SWIR sensor 104 (i.e., after being reflected by objects in the scene).

[0057] SWIR sensor 104 captures SWIR illumination in the SWIR wavelength range produced by SWIR illumination element 106 that passes through filter 108. SWIR sensor 104 generates one or more SWIR images of the scene, e.g., as still images and / or as video (e.g., frames of video).

[0058] Filter 108 is positioned within the field of view (e.g., full field of view) of SWIR sensor 104, and may be located, for example, in front of SWIR sensor 104. SWIR illumination (e.g., generated by SWIR illumination element 106) that reflects off objects in the scene within the field of view passes through filter 108 before reaching SWIR sensor 104. Optionally, all SWIR wavelengths sensed by SWIR sensor 104 first pass through filter 108 before reaching SWIR sensor 104.

[0059] The SWIR illumination element 106 may be positioned behind the filter 108, i.e., the filter 108 is positioned within the illumination field of the SWIR illumination element 106 such that the SWIR illumination generated by the SWIR illumination element 106 passes through the filter 108 toward the scene before being reflected by objects in the scene. Alternatively, the SWIR illumination element 106 may be positioned outside of the filter 108 such that the generated SWIR illumination does not pass through the filter 108 as it is transmitted toward the scene, but instead passes through the filter 108 before reaching the SWIR sensor 104.

[0060] Alternatively or additionally, filter 108 may be an object that is transparent and / or substantially transparent to SWIR illumination and opaque and / or substantially opaque to visible light, such as an LC display. In such an embodiment, SWIR illumination element 106 and filter 108 are positioned behind an LCD display to capture an illuminated SWIR image of a subject's face viewing the LCD display, and analyze the face depicted in the image to perform facial expression recognition while viewing the LCD display.

[0061] Optionally, the system 100 includes one or more of the following components:

[0062] *Hardware processor(s) 110 that can interface with the SWIR sensor(s) 104 to access (e.g., receive) the SWIR image(s). The processor(s) 110 can interface with other components described herein.

[0063] The processor(s) 110 may be implemented as, for example, central processing unit(s) (CPU), graphics processing unit(s) (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), and application specific integrated circuit(s) (ASIC). The processor(s) 110 may include a single processor or multiple processors (homogeneous or heterogeneous) configured for parallel processing, as a cluster, and / or as one or more multi-core processing devices.

[0064] *Memory 112 storing code 112A for execution by processor(s) 110. Code 112A includes program instructions for performing one or more features of the methods described with reference to Figures 2A-2C and 3-10, e.g., analyzing SWIR to recognize facial expressions. Memory 112 may be embodied as, for example, random access memory (RAM), read-only memory (ROM), and / or storage devices, e.g., non-volatile memory, magnetic media, semiconductor memory devices, hard drives, removable storage, and optical media (e.g., DVD, CD-ROM).

[0065] Code 112A may include instructions for detecting and / or extracting facial features in SWIR images and / or for generating other results that may be used by other processes. Code 112A may include instructions for generating a training dataset 114A and / or for training a machine learning model 114B on the training dataset 114A using supervised and / or unsupervised approaches. Examples of ML models include one or more neural networks of various architectures (e.g., artificial, deep, convolutional, fully connected neural networks), support vector machines (SVMs), logistic regression, k-nearest neighbors, decision trees, and combinations of the above.

[0066] *Data storage device(s) 114 that may store data such as, for example, a training dataset 114A including SWIR images of faces and / or extracted facial features labeled with ground truth labels of facial expressions, and / or a trained ML model 114B trained against the training dataset 114A for generating target facial expression results in response to input of target SWIR images of target faces and / or target facial features extracted from the target SWIR images. Data storage device(s) 114 may be embodied as, for example, memory, a local hard disk, virtual storage, a removable storage unit, an optical disk, a storage device, and / or as a remote server and / or computing cloud (e.g., accessed using a network connection).

[0067] *A physical user interface 116 that includes mechanisms for user interaction, for example, to input data (e.g., select a SWIR image for analysis) and / or view data (e.g., determined facial expressions). Exemplary physical user interfaces 116 include, for example, one or more of a touchscreen, a display, a gesture-activated device, a keyboard, a mouse, and voice-activated software using a speaker and microphone. Note that the user interface (e.g., a touchscreen) of the client terminal(s) 122 can be used as the user interface 116.

[0068] *Data interface 118. Data interface 118 communicates with external devices (e.g., server(s) 120 and / or client terminal(s) 122), optionally via network 124, for example, to provide captured SWIR images and / or analysis results of the SWIR images by code 112A, and / or to receive instructions for controlling SWIR sensor(s) 104 and / or processor(s) 110 and / or other components. For example, SWIR images and / or SWIR video captured by SWIR sensor(s) 104 may be streamed (e.g., in real time) via network 124 to server(s) 120 and / or client terminal(s) 122 for remote playback and / or processing.

[0069] The data interface 118 may be implemented as, for example, one or more of a network interface, a USB port, a network interface card, an antenna, a wireless interface for connecting to a wireless network, a short-range wireless connection, a physical interface for connecting to a cable for network connection, a virtual interface implemented in software, network communications software providing a higher layer of network connectivity, and / or other implementations.

[0070] The network 124 may be implemented as, for example, the Internet, a broadcast network, a local area network, a virtual network, a wireless network, a cellular network, a local bus, a point-to-point link (e.g., wired), and / or a combination of the above. Note that a cable connecting the processor(s) 110 to another device may be referred to herein as the network 124.

[0071] It should be noted that one or more of the processor 110, memory 112, data storage device 114, data interface 118, user interface 116, SWIR sensor(s) 104, SWIR lighting element(s) 106, and filter 108 may be part of the computing device 126 and / or may be incorporated into a housing, for example, in a vehicle, behind an LCD in a kiosk, or for installation next to an entrance to a building.

[0072] Computing device 126, server(s) 120, and / or client terminal(s) 122 may be embodied as, for example, one or more and / or combinations of connected devices, client terminals, servers, computing clouds, virtual servers, computing clouds, virtual machines, desktop computers, thin clients, network nodes, network servers, and / or mobile devices (e.g., smartphones, tablet computers, laptop computers, wearable computers, eyeglass computers, and watch computers).

[0073] The server(s) 120 may be implemented as a security server that can analyze the received SWIR image(s) (using code 112A stored in memory associated with each server) and / or analyze the determined facial expressions, for example, to enhance security. For example, a person with a suspicious facial expression may be determined to be a security threat. In another example, the server(s) may be implemented as a content (e.g., advertisements, videos, text, images) server that provides content in response to input of the determined facial expressions. In another example, the server may be implemented as an automated psychotherapy model that provides psychotherapy exercises and / or advice according to input of the determined facial expressions. In another example, the server may be used to monitor a patient (e.g., an in-patient, home-based therapy patient), where, for example, the SWIR images and / or the determined facial expressions are monitored to determine whether the patient is comfortable, in pain, happy, depressed, etc.

[0074] The client terminal(s) 122 may be implemented, for example, as a user's smartphone equipped with a camera that monitors people's facial expressions, cameras connected to computers installed at security checkpoints to check for suspicious facial expressions, and surveillance cameras located around the store that automatically monitor shoppers' facial expressions and generate alerts to employees, such as a distressed shopper who is predicted to need assistance.

[0075] Communication between processor(s) 110 and / or server(s) 120 and / or client terminal(s) 122 via network 124 may be implemented, for example, via application programming interfaces (APIs), software development kits (SDKs), functions and / or libraries and / or add-ons added to existing applications running on server(s) 120 and / or client terminal(s) 122, and / or applications for downloading and execution on server(s) 120 and / or client terminal(s) 122 that communicate with processor(s) 110, function and / or interface calls to code executed by processor(s) 110.

[0076] It should be noted that code 112A may be stored in memory 112 within computing device 126, e.g., for local analysis and / or processing of SWIR images captured by SWIR sensor(s) 104. Alternatively or additionally, one or more portions of code 112A may be stored external to computing device 126, e.g., on memory located in association with server(s) 120 and / or client terminal(s) 122, e.g., SWIR images captured by SWIR sensor(s) 104 are provided over network 124 for remote analysis and / or processing by server(s) 120 and / or client terminal(s) 122.

[0077] Reference is now made to Figure 2A, which is a flowchart of a method for analyzing SWIR images to recognize facial expressions, according to some embodiments of the present invention. One or more features of the methods described with reference to Figures 2A-2C and 3-10 may be implemented by the processor and / or other components described with reference to system 100 of Figure 1.

[0078] At 202, a SWIR illumination pattern and / or a position and / or orientation of SWIR illumination elements may be selected. The SWIR illumination pattern and / or the position and / or orientation of SWIR illumination elements may be selected to create a target illumination pattern on a person's face to generate a SWIR image of the face that is well illuminated. Well illuminating the face can improve the accuracy of automated processes for facial expression recognition, for example, by improving the ability to automatically extract features from the face.

[0079] Examples of target SWIR illumination patterns include pulsating, continuous, polarized, diffuse, and time modulated.

[0080] Examples of positions and / or orientations of the SWIR lighting elements include facing the face, facing from above, facing from below, facing from below, and facing from one side of the face.

[0081] The target SWIR lighting pattern and / or the position / orientation of the lighting elements can be selected to improve illumination of specific facial features. For example, lighting aimed at the front of the face can create a reflective area on the tip of the nose, lighting aimed from above can highlight wrinkles on the forehead, and lighting aimed at the face from the side can highlight pigmentation on the cheeks.

[0082] The target SWIR lighting pattern can be selected to have a luminance that does not induce facial expressions of discomfort. Even at higher luminances and / or when the lighting is directed directly in front of the face, the human eye cannot perceive the SWIR lighting, and therefore, the response to the SWIR lighting will not be depicted in the facial expression. In contrast, a visible light lighting element that produces visible light illumination at the corresponding luminance of the SWIR lighting would normally induce facial expressions of discomfort. For example, when a strong visible light is directed directly in front of the face, a person may express pain on their face, look away from the light, or close their eyes, all of which distort the actual facial expression that would normally be depicted.

[0083] At 204, SWIR illumination, optionally a target SWIR illumination pattern, is generated by SWIR illumination elements, optionally at selected positions and / or orientations.

[0084] SWIR illumination is in a range of wavelengths that are imperceptible / invisible to the human eye.

[0085] It should be noted that other wavelengths imperceptible to the human eye can be used as an alternative to and / or in addition to SWIR.

[0086] Exemplary SWIR illumination is in the wavelength range of approximately 1000-2000 nm.

[0087] Optionally, the SWIR illumination is in the solar blind region of about 1350-1450, or about 1360nm-1380nm.

[0088] One or more filters can be used to filter out non-SWIR and / or non-solar-blind wavelengths and / or to include SWIR and / or solar-blind wavelengths.

[0089] At 206, one or more SWIR image(s) are captured by the SWIR sensor(s). The SWIR sensor(s) capture the SWIR image(s) depicting the face under SWIR illumination.

[0090] Individual still SWIR images can be captured. A sequence of frames of SWIR images, such as a video, can be captured.

[0091] When brighter and / or more direct SWIR illumination is provided, a shorter camera exposure time may be selected and / or a longer camera range may be obtained, and / or finer details may be depicted in the captured SWIR image, as described herein.

[0092] At 208, one or more visible light image(s) captured by a visible light sensor (e.g., standard camera, CMOS, CCD) of the face can be captured. Note that as described herein, low-intensity lighting and / or no artificial lighting is used due to the risk of distorting facial expressions due to external artificial lighting.

[0093] The visible light image may be captured simultaneously with the SWIR image, or may be captured substantially concurrently with the SWIR image (e.g., separated by a time interval short enough so as not to depict changes in facial expression). The visible light camera may be positioned in the same position and / or orientation as the SWIR sensor, or may be positioned in a different position and / or orientation, such as to capture different facial features than those captured by the SWIR sensor.

[0094] At 210, features may be extracted from the SWIR image(s). The features may be, for example, manually created features and / or automatically learned features. The features may be explicitly defined, for example, by a set of rules, templates, patterns, etc. The features may be extracted using feature extraction approaches such as SIFT, HOG, and LBP. The features may not be explicitly defined, for example, extracted by filters and / or weights of a layer of neurons in a neural network. For example, a SWIR image, or a patch thereof, is provided to a trained neural network.

[0095] The features extracted from the SWIR image may be unique features not seen in the visible light image, or may be features that have corresponding features seen in the visible light image, where the features are better seen in the SWIR image than in the visible light image, e.g., providing better contrast to surrounding tissue, more accurate feature detection and extraction, and improved accuracy for facial expression detection.

[0096] Exemplary features are now described. The SWIR image(s) may be analyzed to extract exemplary features, for example, to distinguish each feature from surrounding facial tissue (e.g., skin) for extraction. Features extracted from SWIR images may be more visible (e.g., better highlighted, higher contrast), for example, compared to visible light images. Exemplary features extracted from SWIR images are: Facial hair, such as beards, mustaches, long hair on the top of the head, and sideburns. Makeup for the face, lips, and / or eyes, such as primer, blush, lipstick, foundation, concealer, powder, mascara, etc. Characteristic facial areas that have a higher water content compared to adjacent facial areas with a lower water content. For example, facial areas with a higher water content include the lips, eyeballs, and nasal openings. Facial areas with a lower water content are the skin surrounding the lips, eyeballs, and nasal openings. · Ears distinct from adjacent hair. Tip of the nose. Chin. ·People inside. · Teeth that are distinct from the surrounding lips. Facial lines, for example under the eyes and / or near the nose. Areas of skin that are darker than the surrounding adjacent skin, such as moles, birthmarks, and hyperpigmented areas.

[0097] If visible light image(s) are captured along with the SWIR image, features can also be extracted from the visible light image(s).

[0098] At 212, the SWIR image(s) and / or features extracted from the SWIR image(s) are analyzed to recognize facial expressions depicted by the face.

[0099] Optionally, if a visible light image is captured along with the SWIR image, the visible light image and / or features extracted from the visible light image are analyzed in combination with the SWIR image(s) and / or features extracted from the SWIR image(s).

[0100] Analysis can be performed, for example, by feeding the features and / or images to a trained machine learning model, rule set, etc. Exemplary approaches for training and / or inference with ML models are described herein. Example implementations of ML models include one or more neural networks of various architectures (e.g., convolutional, fully connected, deep, encoder-decoder, regression, graph, neural networks with combinations of multiple architectures), support vector machines (SVMs), logistic regression, k-nearest neighbors, decision trees, boosting, random forests, regressors, etc.

[0101] Optionally, once a sequence of SWIR images has been acquired, the sequence of SWIR images may be analyzed. For example, frames of a SWIR video depicting an initial state of a face changing from an initial state to a facial expression may be analyzed to detect changes from the initial state to the facial expression. Analysis of the sequence may help improve the accuracy of the recognized facial expression; for example, a sequence of features over 1-2 seconds that shows the process of forming a smile is more likely to accurately detect a smile than a single image of a person smiling.

[0102] Optionally, the first frames from the sequence and / or individual images are analyzed to predict facial expressions before the final expression is determined, for example, features indicative of the onset of a smile are detected before the smile appears, allowing a person to predict a smile before it occurs.

[0103] At 214, one or more actions may be automatically triggered in response to the detected facial expression. For example, *SWIR sensors and SWIR lighting elements can be placed in security-sensitive areas such as airport check-ins, bank counters, and entrances to secure areas. They can automatically monitor individuals' faces to detect facial expressions that may indicate a potential security threat, such as nervousness, anger, or drugs. If such facial expressions are detected, an alert can be automatically sent to security personnel before a security breach can be carried out.

[0104] *SWIR sensors and SWIR lighting elements can be placed in stores to automatically monitor an individual's face and detect facial expressions that indicate the person needs assistance, such as confusion, agitation, and anger. Employees can be automatically alerted to assist the person before they reach the point where they need help.

[0105] *SWIR sensors and SWIR lighting elements can be used to monitor patients. The patient's face can be automatically monitored to determine the patient's condition, e.g., comfortable, cold, hot, in pain, depressed, angry, etc. Treatment may be automatically administered accordingly. For example, for a patient determined to be in pain, pain medication can be automatically ordered and / or administered.

[0106] *For example, a user may be automatically monitored while looking at the display, such as watching television or surfing the web. The user's face may be automatically monitored to determine the user's mood. Content may be presented on the display according to the mood. For example, a person who is feeling sad may be shown an advertisement for an antidepressant. In another example, a person who is feeling angry may be shown calming music. In yet another example, a person who is feeling happy may be shown a comedy movie.

[0107] Reference is now made to FIG. 2B, which is a flowchart of a method for training a machine learning model for facial expression recognition using SWIR images, according to some embodiments of the present invention.

[0108] One or more SWIR images depicting the face of the sample individual are acquired at 250. The SWIR images are illuminated using SWIR illumination, optionally with a selected illumination pattern, as described herein.

[0109] At 252, one or more facial expression features may be extracted from the SWIR image(s). Exemplary extractable facial features are described herein.

[0110] One or more visible light images depicting the face of the sample individual can be acquired at 254. The visible light image(s) can be captured simultaneously (e.g., synchronously) or nearly simultaneously (within a time interval) with the SWIR image(s).

[0111] At 256, features can be extracted from the visible light image. The features extracted from the visible light image can correspond to and / or differ from the features extracted from the SWIR image(s).

[0112] At 258, a representation of the facial expression depicted by the sample individual is obtained. The representation may be metadata, text, code, label, etc., such as smiling, sad, angry, crying, laughing, confused, etc.

[0113] At 260, a record is created. The record includes ground truth labels representing facial expressions depicted by the sample individual, and SWIR images and / or facial expression features extracted from the SWIR images. The record may also include visible light images and / or facial expression features extracted from the visible light images.

[0114] At 262, the functionality described with reference to 250-260 can be repeated for different facial expressions and / or for different sample individuals to create multiple records of a multi-record training dataset.

[0115] In H.264, a machine learning model is trained on a training dataset of multiple records. The machine learning model can be implemented, for example, as one or more neural networks of various architectures (e.g., convolutional, fully connected, deep, encoder-decoder, regression, graph, neural networks with a combination of multiple architectures), support vector machines (SVMs), logistic regression, k-nearest neighbors, decision trees, boosting, random forests, regressors, etc. The ML model can be trained using a supervised approach.

[0116] Reference is now made to FIG. 2C, which is a flowchart of a method for analyzing SWIR images to recognize facial expressions, according to some embodiments of the present invention.

[0117] One or more SWIR images depicting the face of the target individual are acquired at 270. The SWIR images are illuminated using SWIR illumination, optionally with a selected illumination pattern, as described herein.

[0118] At 272, one or more facial expression features may be extracted from the SWIR image(s). Exemplary extractable facial features are described herein.

[0119] One or more visible light images depicting the face of the target individual may be acquired at 274. The visible light image(s) may be captured simultaneously (e.g., synchronously) or approximately simultaneously (within a time interval) with the SWIR image(s).

[0120] At 276, features can be extracted from the visible light image. The features extracted from the visible light image can correspond to and / or differ from the features extracted from the SWIR image(s).

[0121] At 278, the target facial expression features extracted from the SWIR image(s) and / or the SWIR image(s) are provided to a machine learning model.

[0122] Optionally, facial expression features of the target extracted from the visible light image(s) and / or the visible light image(s) are fed to the machine learning model in combination with and / or simultaneously with the SWIR image(s) and / or SWIR features.

[0123] The machine learning model can be trained as described with reference to FIG. 2B.

[0124] At 280, a target facial expression for a target individual is obtained as a result of the machine learning model.

[0125] Reference is now made to Figure 3, which is a SWIR image 302A of a person's face under SWIR illumination and a visible light image 302B of the same person, in accordance with some embodiments of the present invention. Figure 3 illustrates that in the SWIR image 302A under SWIR illumination, ears 304A and 306A can be more accurately detected and / or segmented from nearby hair 308A and 310A by an automated process (e.g., a segmentation code, a neural network, a feature detector and / or extractor), thereby enabling the ears to be used as features for facial expression recognition. In contrast, ears 304B and 306B in visible light image 302B are difficult to distinguish from nearby hair 308B and 310B, for example, due to shadows between the ears and the hair and / or similar colors between the ears and the hair.

[0126] Reference is now made to Figure 4, which illustrates another SWIR image 402A of a person's face under SWIR illumination and a visible light image 402B of the same person, according to some embodiments of the present invention. Figure 4 illustrates that in the SWIR image 402A under SWIR illumination, the nose contour 404A, the face line under the eyes 406A, and the nose tip 408A can be more accurately detected and / or segmented by an automated process, thereby enabling these features to be used as features for facial expression recognition. In contrast, the nose contour 404B, the face line under the eyes 406B, and the nose tip 408B in the visible light image 402B are difficult to distinguish from nearby skin, for example, due to facial shadows and / or color similarities between the facial features and the skin.

[0127] Reference is now made to Figure 5, which illustrates yet another SWIR image 502A of a person's face under SWIR illumination and a visible light image 502B of the same person, according to some embodiments of the present invention. Figure 5 illustrates that in the SWIR image 502A under SWIR illumination, features associated with looking up, such as eyebrows 504A, lower eyelids 506A, and pupils 508A, are more accurately detected and / or segmented by an automated process, thereby enabling these features to be used for facial expression recognition. In contrast, the eyebrows 504B, lower eyelids 506B, and pupils 508B in the visible light image 502B are difficult to distinguish from nearby skin, for example, due to facial shadows and / or color similarities between the facial features and the skin.

[0128] Reference is now made to FIG. 6 , which illustrates yet another SWIR image 602A of a person's face under SWIR illumination and a visible light image 602B of the same person, according to some embodiments of the present invention. FIG. 6 illustrates that in the SWIR image 602A under SWIR illumination, there is greater contrast between lips 604A and 606A and teeth 608A, allowing these features to be more accurately detected and / or segmented by automated processes, thereby enabling these features to be used for facial expression recognition. In contrast, there is less contrast between lips 604B and 606B and teeth 608A in the visible light image 602B, making them more difficult to distinguish from each other and / or from nearby skin due to, for example, facial shadows and / or color similarities between the facial features and the skin. Furthermore, in the SWIR image 602A, the outline of nose 610A is much sharper than the outline of nose 610B in the visible light image 602B, allowing automated processes to more accurately detect and / or segment the nasal features.

[0129] Reference is now made to Figure 7, which illustrates yet another SWIR image 702A of a person's face under SWIR illumination and a visible light image 702B of the same person, according to some embodiments of the present invention. Figure 7 illustrates that in the SWIR image 702A under SWIR illumination, facial lines 704A, 706A, 708A, and 710A are more clearly delineated, allowing these features to be more accurately detected and / or segmented by automated processing, thereby enabling these features to be used for facial expression recognition. In contrast, the facial lines 704B, 706B, 708B, and 710B in the visible light image 702B are thinner, making them more difficult to distinguish and / or delineate from nearby skin, for example, due to facial shading and / or color similarities between the facial features and the skin.

[0130] Reference is now made to FIG. 8, which illustrates yet another SWIR image 802A of a person's face under SWIR illumination and a visible light image 802B of the same person, according to some embodiments of the present invention. FIG. 8 illustrates that in the SWIR image 802A under SWIR illumination, skin pigmentation spots 804A, lip borders 806A, nose borders 808A, and inner corners of the eyes 810A are more clearly delineated, allowing these features to be more accurately detected and / or segmented by automated processing, thereby enabling these features to be used for facial expression recognition. In contrast, the skin pigmentation spots 804B, lip borders 806B, nose borders 808B, and inner corners of the eyes 810B in the visible light image 802B are paler, making them more difficult to distinguish and / or delineate from nearby skin, for example, due to facial shading and / or color similarities between the facial features and the skin.

[0131] Reference is now made to FIG. 9, which illustrates yet another SWIR image 902A of a person's face under SWIR illumination and a visible light image 902B of the same person, according to some embodiments of the present invention. FIG. 9 illustrates that in the SWIR image 902A under SWIR illumination, the facial lines 904A, eyes 906A, chin 908A, philtrum 910A, and forehead wrinkles 912A are more clearly defined, allowing these features to be more accurately detected and / or segmented by automated processing, thereby enabling these features to be used for facial expression recognition. In contrast, the facial lines 904B, eyes 906B, chin 908B, philtrum 910B, and forehead wrinkles 912B in the visible light image 902B are more difficult to distinguish and / or delineate from nearby skin due to facial shading and / or color similarities between the facial features and the skin.

[0132] Reference is now made to FIG. 10, which illustrates a sequence of SWIR images 1002A, 1004A, 1006A, 1008A, and 1010A of a person's face under SWIR illumination, captured while smiling, and a sequence of visible light images 1002B, 1004B, 1006B, 1008B, and 1010B of the same person, according to some embodiments of the present invention. SWIR images 1002A-1002A were captured simultaneously with visible light images 1002B-1010B, illustrating a frame-by-frame correspondence. That is, SWIR frame 1004A was captured simultaneously with visible light frame 1004B. Images 1002A-1002B depict the state of the face before the smile motion is initiated. Images 1010A-1010B depict the face while smiling, at the end of the smile motion. The facial features described herein may be easier to detect and / or track in a sequence of SWIR images than in visible light images, for example, to detect the onset of a smile, track the formation of a smile, and predict when a smile is formed (i.e., when a smile ends) and / or to predict a smile in response to detecting the onset of a smile. Based on detecting the onset of a smile, the formation of a smile can be predicted before it occurs. It should be noted that both SWIR and visible light images may be used, for example, to improve the reliability of detection.

[0133] The SWIR image 1002A is labeled with an indication of exemplary facial features that can be used to detect the onset of a smile and / or that can be tracked continuously across frames depicting the formation of a smile. Exemplary facial features include reflections from the outer portions of the cheeks 1050 and / or the bottom of the forehead 1052, which are much weaker and / or imperceptible in visible light images. Another exemplary facial feature includes a bright spot on the tip of the nose 1054, which can be tracked across a sequence of images to detect the small movements of the entire head to the right that occur during a smiling motion. Using these and / or other features, the onset of a smile may be more easily detected in SWIR images than in visible light images, thereby allowing, for example, a smile to be predicted before it appears.

[0134] The description of various embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0135] During the life of the patent that matures from this application, many related SWIR sensors will be developed, and the scope of the term SWIR sensor is intended to include all such new technologies a priori.

[0136] As used herein, the term "about" means ±10%.

[0137] The terms "comprises," "comprising," "includes," "including," "having," and their cognates mean "including, but not limited to." This term encompasses the terms "consisting of" and "consisting essentially of."

[0138] The phrase "consisting essentially of" means that the composition or method may include additional components and / or steps, but only if the additional components and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0139] As used herein, the singular forms "a," "an," and "the" include plural referents unless the context clearly indicates otherwise. For example, the terms "a compound" or "at least one compound" can include multiple compounds, including mixtures thereof.

[0140] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0141] As used herein, the word "optionally" is used to mean "provided in some embodiments and not provided in other embodiments." Any particular embodiment of the present invention may include multiple "optional" features unless such features are inconsistent.

[0142] Throughout this application, various embodiments of the invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all possible subranges and individual numerical values ​​within that range. For example, description of a range such as 1 to 6 should be considered to have specifically disclosed subranges such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, etc., and individual numerical values ​​within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0143] Whenever a numerical range is given herein, it is meant to include any recited number (fractional or integer) within the range given. The phrases "range between" a first reference number and a second reference number and "range" from a first reference number "to" a second reference number are used interchangeably herein and are meant to include the first reference number and the second reference number, and all fractional and integer numbers therebetween.

[0144] It is understood that certain features of the invention that are, for clarity, described in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features of the invention that are, for brevity, described in the context of a single embodiment, can also be provided separately or in any suitable subcombination, or as preferred in any other described embodiment of the invention. Particular features described in the context of various embodiments should not be considered essential features of those embodiments, unless the embodiment cannot function without those elements.

[0145] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0146] It is the intention of the applicant(s) that all publications, patents, and patent applications mentioned herein be incorporated by reference in their entireties, as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference herein. Furthermore, citation or identification of any reference in this application should not be construed as an admission that the reference is available as prior art to the present application. Section headings, if used, should not be construed as necessarily limiting. Additionally, any priority document(s) of this application are incorporated by reference herein in their entireties.

Claims

1. 1. A system for analyzing images and recognizing facial expressions, comprising: At least one processor at least one short wave infrared (SWIR) image representative of a human face depicting a facial expression, the image captured by the at least one SWIR sensor being analyzed; classifying faces in at least one SWIR image into a particular facial expression representing a particular classification category selected from a plurality of predefined facial expressions representing a plurality of classification categories; The system executes code.

2. The system of claim 1, wherein the plurality of predefined facial expressions include joy, sadness, anger, confusion, pain, surprise, fear, and disgust.

3. Extracting a plurality of facial features from at least one SWIR image; The system of claim 1 , wherein the analyzing comprises analyzing a plurality of facial features.

4. The at least one SWIR image includes a single SWIR image; The system of claim 1 , wherein the analysis includes analyzing a single SWIR image, and the classification is performed on the single SWIR image.

5. The system of claim 1, further comprising at least one of: (i) at least one filter that removes electromagnetic radiation of wavelengths that are not significantly absorbed by water vapor in the air; and (ii) at least one shortwave infrared (SWIR) lighting element that generates SWIR lighting for illuminating a person's face.

6. The system of claim 5, wherein the at least one filter includes a spectral narrow passband filter that is one of: (i) a filter that passes wavelengths between about 1350 and 1450 nanometers (nm) and excludes wavelengths greater than about 1450 nm and less than about 1350 nm; and (ii) a filter that passes wavelengths between about 1360 nm and 1380 nm and excludes wavelengths greater than about 1380 nm and less than about 1360 nm.

7. The system of claim 5, further comprising: controlling the SWIR lighting elements to generate a target lighting pattern for illuminating the face; and capturing the at least one SWIR image while the target lighting pattern is being generated.

8. 6. The system of claim 5, wherein the SWIR lighting element produces SWIR illumination at an intensity that does not elicit a facial expression indicative of discomfort, and a visible light lighting element producing visible light illumination at the intensity of the SWIR lighting element elicits a facial expression indicative of discomfort.

9. 6. The system of claim 5, wherein the at least one SWIR lighting element generates illumination in a band of wavelengths invisible to the human eye, and the at least one SWIR sensor senses in the band of wavelengths invisible to the human eye.

10. The system of claim 1, further comprising a visible light image sensor for capturing at least one visible light image, and code for analyzing a combination of the at least one SWIR image and the at least one visible light image.

11. The system of claim 1 , wherein the code further comprises instructions for analyzing the at least one SWIR image to distinguish facial hair on the face from surrounding skin.

12. 10. The system of claim 1, wherein the code further comprises instructions for analyzing at least one SWIR image to distinguish long facial hair on the top of the head from surrounding skin.

13. The system of claim 1 , wherein the code further comprises instructions for analyzing the at least one SWIR image to distinguish makeup on the face from surrounding skin.

14. The system of claim 1, wherein the at least one SWIR image includes a plurality of SWIR frames of a SWIR video depicting an initial state of the face, a transition from the initial state to the facial expression, and the face depicting the facial expression, and the code includes instructions for analyzing the plurality of SWIR frames to detect the transition from the initial state to the facial expression.

15. 2. The system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image to distinguish areas of the face having high water content from adjacent areas of the face having low water content.

16. The system of claim 1 , wherein the code further comprises instructions for analyzing the at least one SWIR image to distinguish ears from adjacent hair.

17. 10. The system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image to detect reflections of SWIR illumination from at least one of a tip of the nose, a chin, and a philtrum.

18. The system of claim 1 , wherein the code further comprises instructions for analyzing the at least one SWIR image to distinguish teeth from surrounding lips.

19. A method for training a machine learning model for recognizing facial expressions, comprising: Creating a training data set of multiple records, the records comprising: at least one SWIR image of the face of the sample individual depicting a facial expression captured by at least one SWIR sensor; and creating a ground truth label indicative of a particular facial expression depicted by the sample individual, the particular facial expression representing a particular classification category selected from a plurality of predefined facial expressions representing a plurality of classification categories; training a machine learning model with the training dataset of multiple records to generate a result of a target facial expression classification category of the target individual selected from the multiple classification categories in response to input of at least one SWIR image of a target face of the target individual under SWIR illumination; The method comprising:

20. 1. A method of analyzing an image for facial expression recognition, comprising: providing at least one SWIR image of a target face of a target individual under SWIR illumination to a machine learning model trained as in the method of claim 19; obtaining a target facial expression classification category of the target individual selected from the plurality of classification categories as a result of the machine learning model; The method comprising:

Images