Shape classification

Abstract

An example computing device may obtain an ambient image with a camera and determine an entropy of the ambient image. Responsive to the entropy satisfying a threshold, the computing device may obtain, with the camera, a first image based on illumination from a first light source; obtain, with the camera, a second image based on illumination from a second light source; generate a ratio image based on the first image, the second image, and the ambient image; and determine a classification for a subject based on the ratio image. Responsive to the entropy not satisfying the threshold, the computing device may obtain the first image; generate a symmetry ratio image based on the first image; and determine the classification for the subject based on the symmetry ratio image. The computing device may perform an action based at least on the classification.

Description

SHAPE CLASSIFICATIONBACKGROUND

[0001] Cameras and light sources for object identification are integrated in computing devices configured for many different applications, such as personal computing devices (e.g., smartphones, wearable computing devices, laptops, desktops, etc.) or computing devices integrated in larger products (e.g., assembly line robotics, cars, bicycles, etc.). Computing device products and / or products with computing devices may include limited space for cameras and light sources used for object identification when considering a desired form factor of the computing device products and / or products with computing devices.SUMMARY

[0002] The techniques of this disclosure enable a computing device to classify a subject using a camera and light sources by dynamically adjusting the number of light source images captured based on the entropy of an ambient image. This approach ensures that the computing device can adapt to varying lighting conditions, thereby improving the accuracy and reliability of subject classification. By determining the entropy of the ambient image, the computing device can decide whether to use one or two light source images, optimizing the classification process for different environmental conditions.

[0003] In cases where the entropy satisfies a threshold, the computing device captures two light source images, which are used to generate a ratio image. This ratio image provides a more detailed representation of the subject's geometry, enhancing the accuracy of the classification. The use of two light sources allows for better differentiation of three- dimensional shapes, which is particularly useful in distinguishing between similar objects or identifying specific features of a subject.

[0004] In cases where the entropy does not satisfy the threshold, the computing device captures a single light source image and generates a symmetry ratio image. This approach simplifies the classification process while still providing sufficient information to accurately classify the subject. The symmetry ratio image is particularly effective in low-light conditions or environments with minimal ambient light, where capturing multiple light source images may not be feasible.

[0005] By adapting to different lighting conditions and dynamically adjusting the number of light source images captured, techniques of this disclosure may ensure that the computing device can maintain high classification accuracy across a wide range of environments. Suchflexibility may enable the techniques of this disclosure to be used for various applications, including facial recognition, object detection, and authentication systems.

[0006] In one example, this disclosure describes a method includes obtaining an ambient image with a camera; determining an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtaining, with the camera, a first image based on illumination from a first light source, obtaining, with the camera, a second image based on illumination from a second light source, generating a ratio image based on the first image, the second image, and the ambient image, and determining a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtaining, with the camera, the first image based on illumination from the first light source, generating a symmetry ratio image based on the first image, and determining the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and performing an action based at least on the classification.

[0007] In another example, this disclosure describes a computing device includes a camera; a first light source; a second light source; at least one processor; and at least one non-transitory computer-readable storage medium storing instructions that are executable by the at least one processor to: obtain an ambient image with the camera; determine an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtain, with the camera, a first image based on illumination from the first light source, obtain, with the camera, a second image based on illumination from the second light source, generate a ratio image based on the first image, the second image, and the ambient image, and determine a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtain, with the camera, the first image based on illumination from the first light source, generate a symmetry ratio image based on the first image, and determine the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and perform an action based at least on the classification.

[0008] In another example, this disclosure describes a non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device to: obtain an ambient image with a camera; determine an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtain, with the camera, afirst image based on illumination from a first light source, obtain, with the camera, a second image based on illumination from a second light source, generate a ratio image based on the first image, the second image, and the ambient image, and determine a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtain, with the camera, the first image based on illumination from the first light source, generate a symmetry ratio image based on the first image, and determine the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and perform an action based at least on the classification.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a conceptual diagram illustrating an example computing device configured to determine a classification for an example subject, in accordance with one or more aspects of the present disclosure.

[0011] FIG. 2 is a block diagram illustrating an example computing device configured to perform classifications of subjects, in accordance with one or more aspects of the present disclosure.

[0012] FIG. 3 is a conceptual diagram illustrating a first example object sensor configuration for classifying an example subject, in accordance with one or more aspects of the present disclosure.

[0013] FIG. 4 is another conceptual diagram illustrating a second example object sensor configuration for classifying an example subject, in accordance with one or more aspects of the present disclosure.

[0014] FIG. 5 is another conceptual diagram illustrating a third example object sensor configuration for classifying an example subject, in accordance with one or more aspects of the present disclosure.

[0015] FIG. 6 is another conceptual diagram illustrating a fourth example object sensor configuration for classifying an example subject, in accordance with one or more aspects of the present disclosure.

[0016] FIG. 7 is a conceptual diagram illustrating an example object sensor for generating a symmetry ratio image, in accordance with one or more aspects of the present disclosure.

[0017] FIG. 8 is a conceptual diagram illustrating example operations of an example computing device configured to generate ratio images for classification of subjects, in accordance with one or more aspects of the present disclosure.

[0018] FIG. 9 is a flowchart illustrating example operations of an example computing device configured to perform actions based on a determined classification of subjects, in accordance with one or more aspects of the present disclosure.DETAILED DESCRIPTION

[0019] FIG. 1 is a conceptual diagram illustrating example computing device 100 configured to determine a classification for example subject 160, in accordance with one or more aspects of the present disclosure. Computing device 100 may include, but are not limited to, user computing devices (e.g., laptops, desktops, and mobile computing devices such as tablets, smartphones, wearable computing devices, artificial intelligence glasses, etc.); embedded computing devices (e.g., devices embedded within a vehicle, camera, image sensor, industrial machine, satellite, gaming console or controller, or home appliance such as a refrigerator, thermostat, energy meter, home energy manager, smart home assistant, etc.); server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, web servers, game servers, application servers, etc.); dedicated, specialized model processing or training devices; virtual computing devices; other computing devices or computing infrastructure; or combinations thereof. In some instances, computing device 100 may represent a cloud computing system that provides one or more services via a network. That is, in some examples, computing device 100 may be a distributed computing system. In some examples, computing device 100 may represent a robotics device, or a component thereof. For example, computing device 100 may be a module or other collection of hardware and / or software components in communication with (e.g., integrated in, wirelessly communicating with, etc.) a robot in a manufacturing assembly line.

[0020] In the example of FIG. 1, computing device 100 may include user interface (UI) components 102, object sensor 104, classification module 110, and application module 114. Although illustrated as internal to computing device 100, UI components 102, object sensor 104, classification module 110, and application module 114 may be distributed at other computing devices or computing systems in communication with computing device 100.

[0021] UI components 102 may include a display and / or input / output (I / O) devices. For example, UI components 102 may include a presence-sensitive display configured to detect input (e.g., touch and non-touch input) from a user operating computing device 100. UI components 102 may output information to a user in the form of a graphical UI, which may be associated with functionality provided by computing device 100. Such UIs may be associated with computing platforms, operating systems, applications, and / or services executing at or accessible from computing device 100 (e.g., identify verification applications or services, shape classification for robotic applications, application implementing human authentication, electronic message applications, chat applications, Internet browser applications, mobile or desktop operating systems, social media applications, electronic games, menus, and other types of applications).

[0022] UI components 102 may receive input, such as tactile, audio, and / or video input. UI components 102, in some examples, include a presence-sensitive display, a fingerprint sensor, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine. UI components 102 may include one or more sensors. Numerous examples of sensors exist and include any input component configured to obtain environmental information about the circumstances surrounding computing device 100 and / or physiological information that defines the activity state and / or physical well-being of a user of computing device 100. In some examples, a sensor may be an input component that obtains physical position, movement, and / or location information of computing device 100. For instance, sensors may include one or more location sensors (e.g., GNSS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., multi-axial accelerometers, gyros), one or more pressure sensors (e.g., barometer), one or more ambient light sensors, and one or more other sensors (e.g., microphone, camera, infrared proximity sensor, hygrometer, and the like). Other sensors may include a heart rate sensor, magnetometer, glucose sensor, hygrometer sensor, olfactory sensor, compass sensor, step counter sensor, to name a few other non-limiting examples.

[0023] UI components 102 may generate one or more outputs. Examples of outputs are tactile, audio, and video output. UI components 102, in one example, includes a presencesensitive display, sound card, video graphics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

[0024] In situations in which the devices or systems discussed here collect personal information about users, or may make use of personal information, the users may be providedwith an opportunity to control whether programs or features collect user information (e.g. , information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current location), or to control whether and / or how to receive content from a content server (e.g., associated with computing device 100) that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by a content server (e.g., associated with computing device 100).

[0025] One or more application modules 114 may include functionality to perform any variety of operations on computing device 100. For instance, application modules 114 may include an operating system, a banking application, a word processor, a text application, a web browser, a multimedia player, a calendar application, a distributed computing application, a graphic design application, a video editing application, a web development application, or any other application that may require shape classification or human authentication (e.g., for facial recognition for standard authentication, presentation attack detection, etc.). In some instances, one of application modules 114 may include an operating system that manages authentication and verification processes associated with providing access to data provided via UI components 102 (e.g., face identity verification for unlocking access to software-based content stored at and / or output by computing device 100). In some examples, one of application modules 114 may include a passport, banking, or other application that requests identity or human confirmation prior to executing an action (e.g., granting access to resources, approving requests input by users, etc.). In some examples, one of application modules 114 may include a robotics application that instructs components of a robot associated with computing device 100 to perform an action (e.g., pick up object, leave object, etc.) based on determined shape classifications. Application modules 114 may process classifications (e.g., shape classification, object classification, animal classifications, plant classifications, etc.) determined by classification module 110 to perform actions specific to applications associated with application modules 114.

[0026] In accordance with the techniques described herein, computing device 100 may determine a classification of subject 160. Subject 160 may include an object (e.g., pipe, box, picture, etc.) or a living being (human, dog, cat, etc.) for which application module 114 hasrequested a classification. Computing device 100, or more specifically classification module 110, may determine a classification of subject 160 based on one or more images captured using object sensor 104. Classification module 110 may include a software module with computer readable instructions for processing data from object sensor 104 to determine a classification of subject 160. Although illustrated as stored at computing device 100, classification module 110 may be stored at another computing device or computing system in communication with computing device 100.

[0027] Classification module 110 may determine a classification of subject 160 according to a classification or identification scheme associated with a particular application of application modules 114. For example, one of application modules 114 may include executable instructions for a software application or service that requests classification of a three- dimensional subject 160. One of application modules 114 may employ or otherwise instruct classification module 110 to determine a shape classification (e.g., cylindrical, cube, a living human face, etc.) of subject 160 based on images captured using object sensor 104.

[0028] Object sensor 104, in the example of FIG. 1, may include one or more cameras 106 and one or more light sources 108. Light sources 108 of object sensor 104 may include any device capable of emitting light that is used to illuminate an environment or scene captured using cameras 106. In some examples, light sources 108 may be capable of generating light that is in the near-infrared range to the infrared range, such that human eyes are insensitive to the light emitted by light sources 108. Light sources 108 may be any suitable light source having a certain degree of spatial and temporal coherence to generate high contrast light patterns. In some examples, light sources 108 may be a laser, such as a super luminescent diode laser. In some examples, light sources 108 may be light emitting diodes (LEDs) or filtered LEDs. In some examples, light sources 108 may be one or more high emissivity diodes, which may be light emitting diodes that are capable of generating light having any range of wavelengths.

[0029] Cameras 106 may include any device for capturing different wavelengths of scenes or environments around computing device 100. For example, Cameras 106 may be any device capable of capturing images of the patterns of light emitted from light sources 108 that are projected onto a three-dimensional scene by computing device 100. That is, cameras 106 may capture the patterns of light that are reflected by the three-dimensional scene. As the light emitted by light sources 108 may have a relatively high wavelength, which may be in the near infrared or infrared range, cameras 106 may be a device that is able to capture the light patterns at such relatively high wavelengths. Cameras 106 may include, for example, aNear-Infrared (NIR) camera, an Infrared (IR) camera, a color or red, blue, green (RGB) camera, an RGB-IR camera, or the like.

[0030] Object sensor 104 may include a camera of camera 106 (hereinafter, “camera 106”) placed adjacent to one or more lights sources 108 on a linear plane. For example, camera 106 may be placed within a center of a housing of object sensor 104. Camera 106 may be placed in the housing such that camera 106 is equidistant from two light sources of light sources 108. Classification module 110 of computing device 100 may obtain images captured using object sensor 104 to generate a ratio image that provides the basis for classification of a subject (e.g., subject 160). By camera 106 being positioned to be adjacent to light sources 108 on a linear plane, classification module 110 may apply various geometric-based equations - based on an environment or scene associated with the subject - to images captured using object sensor 104 to efficiently generate ratio images for classification of subjects. In this way, the orientation of camera 106 and light sources 108 simplifies a three-dimensional problem (e.g., classifying three-dimensional subjects) into two-dimensional equations. Thus, classification module 110 may quickly classify a subject, located in different types of environments, with minimal hardware components.

[0031] In operation, classification module 110 of computing device 100 may obtain an ambient image. Classification module 110 may obtain an ambient image of subject 160 using camera 106 of object sensor 104. An ambient image may be an image that is captured when light sources (e.g., light sources 108) are not activated or otherwise illuminated. Classification module 110 may obtain the ambient image of subject 160 as arrays of pixels representing a scene or environment captured using camera 106.

[0032] Classification module 110 may determine an entropy of the ambient image. Classification module 110 may determine an entropy of pixels in the ambient image. For example, classification module 110 may determine the entropy (e.g., Shannon entropy) as an average of information in the ambient image determined based on a histogram of the ambient image that represents different gray level probabilities of pixels of the ambient image. Classification module 110 may determine the entropy of the ambient image by analyzing a distribution of pixel intensities of the ambient image. Classification module 110 may determine, based on the distribution of pixel intensities, the entropy of the ambient image as an amount of information content or randomness included in the ambient image. Classification module 110 may determine the entropy of the ambient image to provide a context of the environment or scene surrounding subject 160 (e.g., indoor, outdoor, high sunlight exposure, shaded surrounding, etc.).

[0033] Classification module 110 may apply a threshold to entropy values of ambient images to determine whether useful patterns may be ascertained using one or more light sources of light sources 108. For example, in instances where classification module 110 determines that an entropy value of an ambient image satisfies a threshold, classification module 110 may determine that useful information for classification may be obtained with two images captured using two respective light sources of light sources 108. In instances where classification module 110 determines that an entropy value of an ambient image does not satisfy the threshold, classification module 110 may determine that one light source of light sources 108 may provide useful information for generating a ratio image for classification.

[0034] In instances where classification module 110 determines the entropy of the ambient image does not satisfy the threshold, classification module 110 may determine a classification of subject 160 based at least on a symmetry ratio image generated based on a light source image of subject 160 captured using a light source of light sources 108 and an ambient image of subject 160. Classification module 110 may obtain the ambient image of subject 160 with camera 106 while light sources of light sources 108 are not activated, and obtain a light source image of subject 160 while a light source of light sources 108 is activated or is otherwise illuminated.

[0035] In some examples, where the determined entropy of the ambient image of subject 160 does not satisfy a threshold, classification module 110 may generate instructions for object sensor 104 to capture a light source image of subject 160 using one light source of light sources 108. Classification module 110 may generate instructions for object sensor 104 to capture a light source image of subject 160 using a light source of light sources 108 that is positioned to the left with respect to camera 106. In some instances, classification module 110 may generate instructions for object sensor 104 to capture a light source image of subject 160 based on a determined position of an environmental light source (e.g., the sun) relative to computing device 100. For example, classification module 110 may determine the position of an external light source relative to the left of camera 106 based on shadowing included in the ambient image indicating a direction of light rays emitted by the external light source. Based on this determination, classification module 110 may generate instructions for object sensor 104 to capture a light source image of subject 160 using a light source of light sources 108 that is positioned to the right of camera 106.

[0036] Classification module 110 may generate a symmetry ratio image based at least on the light source image captured using the light source of light sources 108. Classification module 110 may generate a symmetry ratio image as an indicator of a geometry of a subject (e.g.,subject 160) captured in the ambient image and the light source image. Classification module 110 may generate a symmetry ratio image based on a difference in image pixel intensities between the light source image and the ambient image divided by the image pixel intensity of the ambient image Classification module 110 may generate the symmetry ratio image as an image that mirrors the light source image captured using the light source of light sources 108.

[0037] Classification module 110 may determine a classification for subject 160 based on symmetry ratio images. Computing device 100 may determine classifications of subject 160, such as a human face classification (e.g., human face or not human face), an object shape classification (e.g., cylinder, cube, etc.), or the like. Classification module 110 may determine the classification for subject 160 by at least providing a symmetry ratio image to one or more machine learning models. For example, classification module 110 may apply a machine learning model that implements segmentation model algorithms and / or geometric classifier algorithms to determine a probability subject 160 should be classified as a particular subject classification (e.g., human face classification, object shape classification, etc.). Classification module 110 may apply the machine learning model to output a score indicating a classification of subject 160. Classification module 110 may determine the classification of subject 160 based on the score.

[0038] In instances where classification module 110 determines the entropy of the ambient image satisfies a threshold, classification module 110 may determine a classification of subject 160 based at least on the ambient image and a ratio image generated based on two or more light source images captured using light sources of light sources 108. Classification module 110 may obtain the two or more light source images of subject 160 using camera 106 and two or more respective light sources of light sources 108. For example, classification module 110 may obtain a first light source image of subject 160 based on illumination from a first light source of light sources 108. Classification module 110 may obtain a second light source image of subject 160 based on illumination from a second light source of light sources 108.

[0039] Classification module 110 may generate a ratio image based on the ambient image and light source images captured using light sources 108. Classification module 110 may apply geometric-based equations to the ambient image and the light source images captured using light sources 108. Classification module 110 may generate a ratio image based on a difference in image pixel intensity between a first light image taken with a first light source of light sources 108 and a second light source image taken with a second light source of lightsources 108, where an ambient image is subtracted from both the first light source image and the second light source image.

[0040] Classification module 110 may determine a classification for subject 160 based on one or more ratio images. Computing device 100 may determine classifications of subject 160, such as a human face classification (e.g., human face or not human face), an object shape classification (e.g., cylinder, cube, etc.), or the like. Classification module 110 may determine the classification for subject 160 by at least providing a ratio image to one or more machine learning models. For example, classification module 110 may apply a machine learning model that implements segmentation model algorithms and / or geometric classifier algorithms to determine a probability subject 160 should be classified as a particular subject classification (e.g., human face classification, object shape classification, etc.). Classification module 110 may apply the machine learning model to output a score indicating a classification of subject 160. Classification module 110 may determine the classification of subject 160 based on the score.

[0041] Computing device 100 may perform an action based on the determined classification of subject 160. For example, computing device 100 may perform an action of selectively unlocking access to a user interface output by computing device 100 based at least on the determined human face classification of subject 160 indicating that subject 160 is associated with a real living, pre-registered human face of a user operating computing device 100. Computing device 100 may apply classification module 110, as well as other techniques for face recognition, to determine that subject 160 is a real human face, and that the real human face is registered as having access to resources associated with the operation of computing device 200. In another example, computing device 100 may perform an action of generating instructions for a robot to interact with subjects (e.g., subject 160) based on a determined classification of the objects (e.g., instructions for the robot to pick up objects classified as cylindrical). Computing device 100 may output an identification of the subject based at least on the classification specifying an object type (e.g., pipe, box, human, etc.) associated with the subject.

[0042] The techniques may provide one or more technical advantages that realized a practical application. For example, computing device 100 may process images captured by object sensor 104 to determine a classification of a subject with a higher degree of confidence. The techniques described herein implement two-dimensional measurements to classify three- dimensional subjects with high accuracy without a need to compute a depth of subjects, which is a goal of photometric stereo techniques that include three or more light sources orilluminators. By applying particular geometric based equations based on an environment associated with the subject, computing device 100 may classify the subject throughout various types of environments or scenes (e.g., indoor, outdoor, etc.). Conventionally, techniques may include Vertical Cavity Surface Emitting Lasers (VCSELs) as lasers for three dimensional classifications. However, VCSELs may be expensive, may need special circuits to make them safe for user operation, and / or may have a thickness that prevents installation in thin panels within a computing device. The techniques described herein may implement small, low-cost light sources (e.g., LEDs) for three-dimensional subject classification using two-dimensional measurements. In this way, the techniques described herein may improve the form factor of and / or reduce operational complexity of computing devices tasked with subject classification, while improving robustness of subject classification throughout various environments.

[0043] FIG. 2 is a block diagram illustrating an example computing device configured to perform classifications of subjects, in accordance with one or more aspects of the present disclosure. In the example of FIG. 2, computing device 200 may include one or more processors 220, one or more communication units 226, user interface device 202, object sensor 204, and storage devices 228. Computing device 200, user interface (UI) device 202, object sensor 204, cameras 206, light sources 208, classification module 210, and application modules 214 of FIG. 2 may be example or alternative implementations of computing device 100, UI components 102, object sensor 104, cameras 106, light sources 208, classification module 110, and application module 114 of FIG. 1, respectively. Communication channels 250 may interconnect each of components 220, 226, 202, 204, and / or 228 for intercomponent communications (physically, communicatively, and / or operatively). In some examples, communication channels 250 may include a system bus, a network connection, one or more inter-process communication data structures, or any other components for communicating data between hardware and / or software.

[0044] One or more communication units 226 may communicate with external devices by transmitting and / or receiving data. For example, computing device 200 may use communication units 226 to transmit and / or receive radio signals and radio networks such as a cellular radio network. In some examples, communication units 226 may transmit and / or receive satellite signals on a satellite network such as a Global Positioning System (GPS) network. Examples of communication units 226 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples ofcommunication units 226 include Bluetooth®, GPS, 3G, 4G, and Wi-Fi® radios found in mobile devices as well as Universal Serial Bus (USB) controllers and the like.

[0045] One or more processors 220 may implement functionality and / or execute instructions with computing device 200. For example, processors 220 on computing device 200 may receive and execute instructions stored by storage devices 228 that provide the functionality of classification module 210, operating system (OS) 212, and / or application modules 214, for example. These instructions executed by processors 220 may cause computing device 200 to store and / or modify information, within storage devices 228 during program execution.

[0046] One or more storage devices 228 within computing device 200 may store information for processing during operation of computing device 200. In some examples, storage devices 228 are a temporary memory, meaning that a primary purpose of storage devices 228 is not long-term storage. Storage devices 228 of computing device 200 may be configured for shortterm storage of information as volatile memory and therefore not retain stored contents if deactivated. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

[0047] Storage devices 228, in some examples, also include one or more computer-readable storage media. Storage devices 228 may be configured to store larger amounts of information than volatile memory. Storage devices 228 may further be configured for long-term storage of information as non-volatile memory space and retain information after on / off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices 228 may store program instructions and / or data associated with classification module 210, operating system (OS) 212, and / or application modules 214.

[0048] OS 212 may control the operation of computing device 200. For example, OS 212 may facilitate the communication of classification module 210, application modules 214, ambient images 242, light source images 244, object classifications 246, training data, and OS 212 with processors 220, storage devices 228, communication units 216, UI device 202, and object sensor 204. In some examples, OS 212 may manage interactions between software applications and a user operating computing device 200. OS 212 may have a kernel that facilitates interactions with underlying hardware of computing device 100 and provides a fully formed application space capable of executing a wide variety of software applicationshaving secure partitions in which each of the software applications executes to perform various operations.

[0049] In the example of FIG. 2, classification module 210 may include entropy module 232, ratio images module 234, and machine learning models 236. Entropy module 232 may include software readable instructions for determining an entropy of ambient images stored as ambient images 242. Ambient images 242 may store ambient images captured using cameras 206 of object sensor 204. For example, application modules 214 may request that classification module 210 classify a subject. Entropy module 232 of classification module 210 may generate instructions that cause camera 206 of object sensor 204 to capture an image of a scene including an object without any light sources of light sources 208 activated or otherwise illuminated. Entropy module 232 may store the image captured using cameras 206 as ambient images 242.

[0050] Entropy module 232 may process ambient images in ambient images 242 to determine an entropy of the ambient images. For example, entropy module 232 may process an ambient image stored at ambient images 242 to measure an entropy of image pixels included in the ambient image. Entropy module 232 may process the ambient image by defining a subject detection box that reduces the ambient image to a portion associated with a subject to be classified (e.g., a subject detection box representing a portion of an ambient image including subject 160 of FIG. 1). Entropy module 232 may measure the entropy of image pixels in the subject detection box as an average of information in the image pixels of the subject detection box determined based on a histogram of the subject detection box that represents different gray level probabilities of the image pixels. Entropy module 232 may, based on the histogram, the entropy of the ambient image as an amount of information content or randomness included in the ambient image. In this way, entropy module 232 may determine an amount of information available during a particular classification process under particular conditions. Entropy module 232 may send an entropy of an ambient image to ratio images module 234.

[0051] Ratio images module 234 may include computer readable instructions for generating ratio images based on an ambient image of ambient images 242 and one or more light source images of light source images 244. Light source images 244 may store light source images captured using cameras 206 and light sources 208. Ratio images module 234 may generate instructions that cause cameras 206 to capture a light source image while one light source of light sources 208 is activated or otherwise illuminated. Ratio images module 234 may generate instructions for capturing light source images based on the entropy of an ambientimage obtained from entropy module 232. For example, ratio image module 234 may determine that an entropy value of an ambient image of a subject satisfies a threshold (e.g., the entropy of an ambient image is greater than an entropy threshold). Ratio image module 234 may generate instructions for object sensor 204 to capture a first light source image of the subject using cameras 206 and a first light source of light sources 208. Ratio image module 234 may generate instructions for object sensor 204 to capture a second light source image of the subject using cameras 206 and a second light source of light sources 208. Ratio image module 234 may generate a ratio image of the subject based on the ambient image of the subject, the first light source image of the subject, and the second light source image of the subject.

[0052] In another example, ratio image module 234 may determine that an entropy value of an ambient image of a subject does not satisfy a threshold (e.g., the entropy of an ambient image is less than or equal to an entropy threshold). Ratio image module 234 may generate instructions for object sensor 204 to capture a light source image of the subject using cameras 206 and a light source of light sources 208 (e.g., the left-most light source of light sources 208). Ratio image module 234 may generate a symmetry ratio image based on the light source image. Ratio image module 234 may generate the symmetry ratio image by computing a symmetric ratio of a subject identified in the light source image. Ratio image module 234 may generate the symmetry ratio image as an indicator of a geometry of a subject, which is distinct based on different subject classifications. For example, a symmetry ratio image indicating geometry associated with a human face may be distinct from symmetry ratio images indicating geometries of objects (e.g., pictures, pipes, boxes, etc.).

[0053] Ratio images module 234 may generate a ratio image and / or symmetry ratio images using one or more machine learning models of machine learning models 236. Machine learning models 236 may include software modules for implementing computer vision techniques (e.g., shape from shading techniques, facial recognition techniques, etc.), segmentation models, neural networks (e.g., convolutional neural networks, deep neural networks, etc.), geometry classifiers, support vector machine (SVM) algorithms, linear regression algorithms, clustering algorithms, classification algorithms, or the like. Ratio images module 234 may apply a first machine learning model of machine learning models 236 generate a ratio image that represents a shape of a subject depicted in an ambient image and corresponding one or more light source images.

[0054] In some instances, motion may be detected with respect to a subject depicted in an ambient image relative to the subject depicted in one or more light source images. In thisinstance, ratio images module 234 may compute the difference and ratio of the ambient image and the light source images and the light source images using post-thumbnailing techniques (e.g., random translations of thumbnails as augmentations). In some examples, ratio images modules 234 may apply one or more convolutional layers of a machine learning model of machine learning models 236. Ratio images module 234 may augment ambient images and light source images using convolutional layers that share weights prior to subtraction or division of the ambient image and the light source images. Ratio images module 234 may apply at least two convolutional layers (e.g., a first convolutional layer having sixteen 5x5 filters with stride 2 and a second convolutional layer having thirty -two 3x3 filters) of the machine learning model of machine learning models 236 to preprocess ambient images and light source images. Ratio images module 234 may empirically set the number of filters and architecture of the convolutional layers used to preprocess ambient images and light source images.

[0055] Classification module 210 may apply a machine learning model of machine learning models 236 (e.g., a shape classification machine learning model) trained to determine object classifications of object classifications 246 based on ratio images and / or symmetry ratio images. For example, classification module 210 may apply a classification model of machine learning models 236 that includes a depth estimation model or a shading analysis model for light source images. The classification model of machine learning models 236 may apply depth estimations of light source images to perform geometry classification to determine probabilities a subject belongs to a particular classification (e.g., face shape, non-face shape, cylinder, cube, etc.).

[0056] Classification module 210 may provide ratio images, light source images, and / or symmetry ratio images to machine learning models 236 to classify a subject. Machine learning models 236 may include a first machine learning model trained to determine a texture of the subject. Machine learning models 236 may include a second machine learning model trained to determine a shape of the subject. Machine learning models 236 may include a third machine learning model trained to determine a consistency of light source images. Machine learning models 236 may include a fourth machine learning model trained to determine one or more facial features of the subject. Machine learning models 236 may include a fifth machine learning model trained to determine an indicative area (e.g., subject detection box) of a scene associated with the ambient image. Machine learning models 236 may include a sixth machine learning model trained to determine one or more occlusions in a scene associated with the subject. Machine learning models 236 may classify the subjectbased at least one of the output from the various machine learning models, such as a texture of the subject, the shape of the subject, the consistency of the light source images, one or more facial features of the subject, a subject detection box, and one or more occlusions in the scene associated with the subject.

[0057] Machine learning models 236 may determine a classification of a subject based on object classifications 246. Object classifications 246 may store a mapping of object classifications to actions defined by OS 212 and / or application modules 214. For example, in instances where an application module of application modules 214 requests classification module 210 to classify whether a subject is a living human or not, object classifications 246 may store a first mapping of scores corresponding to a first classification of not human face to a first action of outputting a negative result and a second mapping of scores corresponding to a second classification of a human face to a second action of outputting a positive result or unlocking access to a resource. Computing device 200 may perform an action (e.g., granting access to a resource, interacting with a subject, etc.) specified in object classifications 246 based on the determined classification of a subject.

[0058] Machine learning models 236 may be trained to perform particular tasks (e.g., augmenting light source images and ambient images, subject classification based on ratio images, etc.) based on training data 248. Machine learning model 236 may be trained to classify subjects based at least on training data including ratio images or symmetry ratio images of known subjects generated based on images taken using object sensor 204. Training data 248 may include example ratio images with corresponding example ambient images and light source images. Training data 248 may include labels for each example ratio image indicating a ground truth (e.g., a ground truth specifying an actual classification of a subject depicted in the example images). Training data 248 may include images with varying degrees of entropy values to train machine learning models 236 to be robust throughout different types of environments.

[0059] FIG. 3 is a conceptual diagram illustrating a first example object sensor configuration for classifying example subject 360, in accordance with one or more aspects of the present disclosure. Subject 360, computing device 300, camera 306, and light sources 308A, 308B of FIG. 3 may include example or alternative implementations of subject 160, computing device 100, cameras 106, and light sources 108 of FIG. 1, respectively.

[0060] In the example of FIG. 3, an object sensor (e.g., object sensor 104 of FIG. 1) of computing device 300 may include camera 306, light source 308 A, and light source 308B. The orientation of camera 306, light source 308 A, and light source 308B may form a baseline(“b”). That is, camera 306 is located a distance of a times the baseline from light source 308 A, and a distance of (1-a) times the baseline from light source 308B. Subject 360 may be placed within a scene such that subject 360 is located a distance “d” from computing device 300. Subject 360 may be placed within a scene or environment such that a point on subject 360 has a normal vector (“n”) with respect to a point on subject 360, a first vector (“vl”) with respect to light source 308 A, and a second vector (“v2”) with respect to light source 308B. First vector vl may correspond to radiation or rays subtended by light source 308A at a point of subject 360 (represented as normal vector “n”) when capturing a first light source image. Second vector v2 may correspond to radiation or rays subtended by light source 308B at the point of subject 360 when capturing a second light source image. Subject 360 may be placed in a scene such that a subject detection box of subject 360 is defined using the angle 0. In other words, subject 360 in the scene is measured by the angle 0 that the point subject 360 makes with a center of camera 306. According to the orientation illustrated in FIG. 3, the three dimensional problem of classifying a three dimensional subject 360 is simplified to two dimensional, geometric-based equations.

[0061] Computing device 300 may generate, based on images captured using camera 306 and light sources 308A, 308B, ratio images for determining a classification of subject 360. In the example of FIG. 3, computing device 300 may determine entropy of an ambient image of subject 360 satisfies a threshold. Computing device 300 may capture a first light source image using camera 306 and light source 308 A, and a second light source image using camera 306 and light source 308B. Computing device 300 may determine an image intensity of the first light source image based at least on a proportionality between the image intensity and a dot product of vector vl and the normal vector n. Computing device 300 may determine an image intensity of the second light source image based at least on a proportionality between the image intensity and a dot product of vector v2 and the normal vector n. Computing device 300 may process the first light source image and the second light source image by scaling the respective image intensities such that the maximum value of image intensity is a predefined value (e.g., “255”). Computing device 300 may generate a ratio image by taking a difference between the scaled first light source image and the scaled second light source image.

[0062] In general, computing device 300 may generate a ratio image with image processing techniques (e.g., a difference function that subtracts pixel values of one image from corresponding pixel values of another image, a ratio function that divides pixel values of one image by corresponding pixel values of another image to compare image intensities, abackground subtraction function that detects objects and / or motion of objects, etc.) based on baseline b, distance d, and vectors resulting from a placement of subject 360. Computing device 300 may apply a classification model (e.g., deep neural networks designed to train classifiers for classifying subjects in a particular application) to determine a classification of a subject based on the ratio image. In this way, computing device 300 may classify subjects in low light environments or scenes using a single near-infrared camera and two, low-cost light sources (e.g., LEDs), each having a small form factor.

[0063] FIG. 4 is another conceptual diagram illustrating a second example object sensor configuration for classifying example subject 460, in accordance with one or more aspects of the present disclosure. Subject 460, computing device 400, camera 406, and light sources 408 A, 408B of FIG. 4 may include example or alternative implementations of subject 160, computing device 100, cameras 106, and light sources 108 of FIG. 1, respectively.

[0064] In the example of FIG. 4, the orientation of camera 406 and light sources 408 A, 408B with respect to subject 460 may result in a first vector (“VI”) with light source 408 A, a second vector (“V2”) with light source 408B, and a third vector (“V3”) with camera 406. The vector VI may correspond to radiation or rays subtended by light source 408 A at a point of subject 460 when camera 406 captures a first light source image. The vector V2 may correspond to radiation or rays subtended by light source 408B at the point of subject 460 (represented as normal vector “n”) when camera 406 captures a second light source image. The vector V3 may correspond to a reflectance value associated with rays reflected by a point of subject 460 and captured by camera 406.

[0065] Computing device 400 may generate a ratio image based on vectors VI, V2, and V3. For example, computing device 400 may capture an ambient image of subject 460 using camera 406. Computing device 400 may that determine an entropy value of the ambient image satisfies a threshold. Computing device 400 may generate instructions to capture a first light image of subject 460 using light source 408 A and camera 406. Computing device 400 may generate instructions to capture a second light source image of subject 460 using light source 408B and camera 406. Computing device 400 may compute image thumbnails for each of the first light source image, the second light source image, and the ambient image based on vectors VI, V2, and V3, respectively.

[0066] In instances where subject 460 is not moving, computing device 400 may calculate the difference between an image thumbnail corresponding to the first light source image and an image thumbnail corresponding to the ambient image. Computing device 400 may similarly calculate the difference between an image thumbnail corresponding to the secondlight source image and the image thumbnail corresponding to the ambient image. Computing device 400 may calculate the differences between image thumbnails corresponding to the light source images and the ambient image. For example, computing device 400 may take the difference between an image thumbnail of the first light source image and an image thumbnail of the ambient image, and take the difference between an image thumbnail of the second light source image and the image thumbnail of the ambient image. Computing device 400 may calculate the difference between image thumbnails corresponding to the light source images and the ambient image using computer vision functions that subtract pixel values of the ambient image from pixel value of the light source images.

[0067] Computing device 400 may generate a ratio image by computing a ratio of the differences of the light source images and the ambient image. Computing device 400 may generate the ratio image based on the ratio such that the ratio includes a term proportional to a ratio of the cosine of the angles 0. Computing device 400 may compute the ratio based on the cosine of angles 0 representing angles (represented as vectors On of VI and 0i2 of V2) of rays emitted by light source 408A and light source 408B subtended with respect to a point of subject 460 (represented as the normal vector n), and an angle (represented as vector Or of V3) of a reflectance value captured using camera 406 with respect to the point of subject 460.

[0068] Computing device 400 may generate a ratio image that is representative of the shape of subject 460 based on the geometric proportionality represented by the ratio of the difference between the first light source image and the ambient image and the difference between the second light source image and the ambient image. Computing device 400 may apply a classifier model, trained to classify particular subjects (e.g., subject 460) based on ratio images, to determine a classification of the subject based on the ratio image. Computing device 400 may perform an action based on the determined classification, such as provide access to resources based on a classification that subject 460 is a living human face or interact with subject 460 based on a classification that subject 460 is a cube.

[0069] In instances where subject 460 may be in motion or otherwise moving, the difference between the first light source image and the ambient image and the difference between the second light source image and the ambient image may lead to inaccuracies. Computing device 400 calculating the difference between image thumbnails of the light source images and the ambient image may improve accuracy by including a value to each vector representing motion of subject 460 (e.g., represents a relative phase shift associated with each vector). In some instances, computing device 400 may apply random translations of the image thumbnails of the light source images and the ambient image as augmentations tocompensate for the motion of subject 460. In some examples, computing device 400 may augment the light source images and the ambient image using convolutional layers of a machine learning algorithm prior to calculating the difference and the ratio. For example, computing device 400 may generate a ratio image of subject 460 using one or more machine learning models (e.g., convolutional neural networks) with convolutional layers that share weights when processing the ambient image and the light source images. Computing device 400 may apply the convolutional layers to process the light source images and ambient image. Computing device 400 may generate the ratio image by subtracting the log of the difference of a processed, intermediate version of the first light source image and a processed, intermediate version of the ambient image, from the log of the difference of a processed, intermediate version of the second light source image and the processed, intermediate version of the ambient image.

[0070] FIG. 5 is another conceptual diagram illustrating a third example object sensor configuration for classifying an example subject, in accordance with one or more aspects of the present disclosure. Subject 560, computing device 500, camera 506, and light source 508 of FIG. 5 may include example or alternative implementations of subject 160, computing device 100, cameras 106, and light sources 108 of FIG. 1, respectively.

[0071] In the example of FIG. 5, computing device 500 may include camera 506 and light source 508. Computing device 500, in the example of FIG. 5, may determine that an entropy of an ambient image of subject 560, captured using camera 506, satisfies a threshold. For example, computing device 500 may determine that a determined entropy value of pixels associated with an ambient image of subject 560 indicates a pattern of pixel intensity distribution. Computing device 500, in the example of FIG. 5, may determine, based on the ambient image, that subject 560 is illuminated by an environmental light source (e.g., the sun, street lamps, ceiling light, etc.). Computing device 500 may determine subject 560 is illuminated by the environmental light source based on computer vision analysis techniques (e.g., illumination estimation based on shadow detection or specular highlights, photometric stereo techniques, reflectance models, shading models, pre-trained machine learning models, shape-from-shadow algorithms, etc.). Computing device 500 may determine an environmental light source direction with respect to subject 560 using the computer vision analysis techniques. Based on computing device 500 determining the entropy value of the ambient image satisfies a threshold and determining the direction of the environmental light source, computing device 500 may generate a ratio image for classifying subject 560 based on a single light source image captured using camera 506 and light source 508.

[0072] Computing device 500 may determine a first vector (“VI”) representing the direction rays emitted by the environmental light source are subtended with respect to a point of subject 560 (represented as normal vector “n”) based on the ambient image. Computing device 500 may determine a second vector (“V2”) representing radiation or rays subtended by light source 508 at the point of subject 560 when capturing the light source image. Computing device 500 may determine a third vector (“V3”) representing a reflectance value of a point of subject 560 captured by camera 506. In this way, computing device 500 may use the environmental light source as a second illuminator to apply techniques similar to those described in FIG. 4, for example.

[0073] Computing device 500 may generate a ratio image based on vectors VI, V2, and V3, as illustrated in FIG. 5. For example, computing device 500 may generate the ratio image based on a ratio of the difference between the light source image and the ambient image over the ambient image. In some instances, computing device 500 may create image thumbnails for the light source image and the ambient image. Computing device 500 may generate the ratio image based on a ratio of the difference between a first image thumbnail associated with the light source image and a second image thumbnail associated with the ambient image over the second image thumbnail associated with the ambient image. Computing device 500 may take a difference and ratio of images using computer vision functions that perform operations (e.g., subtraction, division, etc.) to pixel value of the images.

[0074] FIG. 6 is another conceptual diagram illustrating a fourth example object sensor configuration for classifying example subject 660, in accordance with one or more aspects of the present disclosure. Subject 660, computing device 600, camera 606, and light source 608 of FIG. 6 may include example or alternative implementations of subject 160, computing device 100, cameras 106, and light sources 108 of FIG. 1, respectively.

[0075] In the example of FIG. 6, computing device 600 may include camera 606 and light source 608. Computing device 600, in the example of FIG. 6, may determine that an entropy of an ambient image of subject 660, captured using camera 606, does not satisfy a threshold. For example, computing device 600 may determine that a determined entropy value of pixels included an ambient image of subject 560 does not indicate a pattern of pixel intensity distribution. Computing device 600, in the example of FIG. 6, may determine, based on the entropy of the ambient image and / or an application of computer vision analysis techniques, that there is not an environmental light source that can be used as a second illuminator for generating ratio images used in classifying subject 660. For example, computing device 600 may determine subject 660 is in a dark environment (e.g., outdoors during nighttime) or is ina very bright environment (e.g., outdoors with harsh sunlight that may wash out an image) such that a direction of an environmental light source may not be ascertained. In this instance, computing device 600 may generate a ratio image for classifying subject 660 based on a single light source image of subject 660 captured using light source 608 and camera 606, a symmetry ratio image generated based on the light source image, and an ambient image of subject 660 captured using camera 606. In this way, computing device 600 may generate the symmetry ratio image to compensate for a second light source image that may be used to generate a ratio image, as previously described.

[0076] Computing device 600 may generate a symmetry ratio image based on the light source image captured using light source 608 and camera 606. Computing device 600 may generate the symmetry ratio image based on vectors representing an orientation of light source 608 and camera 606 with respect to subject 660. Computing device 600 may generate the symmetry ratio image based on a first vector (“vl”) representing rays subtended by light source 608 at a point of subject 660 (represented as normal vector “n”) when capturing the light source image and a second vector (“v2”) representing a reflectance value at the point of subject 660 captured using camera 606. Computing device 600 may define vector vl to include a first angle value (“Oil”) representing an angle between a ray of light source 608 subtending a point on subject 660 and a normal vector n at the point on subject 660. Computing device 600 may define vector v2 to include a second angle value (“Or”) representing an angle between a direction corresponding to a reflectance value at the point on subject 660 captured by camera 606 and the normal vector n. In some instances, computing device 600 may include a value ii in the first vector vl and a value r in the second vector v2 to account for motion of subject 660, where values n and r may represent relative phase shifts. Computing device 600 may generate the symmetry ratio image by applying computer vision techniques for generating self-ratio images (e.g., symmetry evaluation frameworks, shape from shading techniques, etc.) given vectors vl and v2. Computing device 600 may determine a classification of a subject based on applying a classification model to process the symmetry ratio image.

[0077] FIG. 7 is a conceptual diagram illustrating an example object sensor for generating a symmetry ratio image, in accordance with one or more aspects of the present disclosure. Subject 760 and camera 706 of FIG. 7 may include example or alternative implementations of subject 160 and cameras 106 of FIG. 1, respectively. FIG. 7 may be discussed with respect to FIG. 1 for example purposes only.

[0078] Computing device 100, in the example of FIG. 7, may determine that an ambient image captured using camera 706 does not satisfy a threshold. Computing device 100 may generate a symmetry ratio image based on a light source image captured using a light source of light sources 108 and camera 706. Computing device 100 may generate the symmetry ratio image based on the light source image by defining vectors (“r”) representing a ray (e.g., vector rl) emitted by the light source of light sources 108 and a shifted version of the ray (e.g., vector r2) at another end of an image plane of camera 706 defined by point “u” and point “v”. Computing device 100 may determine vector tl based on a reflectance associated with the rays of vector rl. Similarly, computing device 100 may determine vector t2 based on a reflectance associated with the rays of vector r2.

[0079] Computing device 100 may generate a symmetry ratio image based on vectors rl, r2, tl, and t2. For example, computing device 100 may generate the symmetry ratio image based on a ratio of the cosine of vector rl minus vector tl over the cosine of vector r2 minus vector t2. Computing device 100 may provide the symmetry ratio image to a geometry classification model trained to classify subjects. Computing device 100 may apply the geometry classification model to classify subject 760 based on the symmetry ratio image.

[0080] FIG. 8 is a conceptual diagram illustrating example operations of example computing device 800 configured to generate ratio images for classification of subjects, in accordance with one or more aspects of the present disclosure. Machine learning models 836A and machine learning models 836B may be example implementations or alternative implementations of machine learning models 236 of FIG. 2. FIG. 8 may be discussed with respect to FIG. 1 for example purposes only.

[0081] In the example of FIG. 8, machine learning model 836A may include convolutional layers 856A, 856B, and 856C (collectively referred to herein as “convolutional layers 856”). For example, machine learning model 836A may include a machine learning model with a convolutional neural network architecture. Machine learning model 836Amay include convolutional layers 856 with filters and architectures based on a shape classification model (e.g., classification model 858) integrating machine learning model 836Afor preprocessing images. For example, machine learning model 836A may include convolutional layers 856 with two layers. In instances where classification model 858 includes a classification model trained to classify whether a subject is a human, convolutional layers 856 may include a first layer having sixteen 5x5 filters with a stride 2, and a second layer having thirty -two 3x3 filters.

[0082] Computing device 800, in the example of FIG. 8, may determine that a subject to be classified is in motion. For example, computing device 800 may determine that there is a relative shift in position of a subject as depicted in ambient image 842, light source image 844A, and / or light source image 844B. Computing device 800 may generate ambient image 842 using camera 106. Computing device 800 may generate light source image 844A using a first light source of light sources 108 and camera 106. Computing device 800 may generate light source image 844B using a second light source of light sources 108 and camera 106. Computing device 800 may apply convolutional layers 856Ato generate a first immediate representation by processing light source image 844A, convolutional layers 856B to generate an ambient intermediate representation by processing ambient image 842, and convolutional layers 856C to generate a second intermediate image by processing light source image 844B. Computing device 800 may apply computer vision functions to subtract the ambient intermediate representation from the first intermediate representation. Computing device 800 may apply computer vision functions to take log of the difference between the first intermediate and the ambient intermediate representation. Similarly, computing device 800 may apply computer vision functions to subtract the ambient intermediate representation from the second intermediate representation. Computing device 800 may apply computer vision functions to take the log of the difference between the second intermediate representation and the ambient intermediate representation. Computing device 800 may subtract the log of the differences and provide the output to classification model 858. Classification model 858 may output, based on the output of the preprocessing, a classification of a subject using a classification machine learning model (e.g., a convolutional neural network trained to classify a subject in a ratio image as a human or not human).Computing device 800 may apply operations depicted in FIG. 8 in instances where no motion is detected to increase the robustness of subject classification.

[0083] FIG. 9 is a flowchart illustrating example operations of an example computing device configured to perform actions based on a determined classification of subjects, in accordance with one or more aspects of the present disclosure. FIG. 9 may be described with respect to FIG. 1 for example purposes only.

[0084] Computing device 100 may obtain an ambient image of subject 160 (902). Computing device 100 may obtain the ambient image of subject 160 using camera 106. Computing device 100 may determine an entropy of the ambient image (904). Computing device 100 may determine the entropy of the ambient image based on an analysis of a distribution of pixel intensities of the ambient image. Computing device 100 may determinethe entropy of the ambient image as an amount of information content or randomness included in the ambient image. Computing device 100 may determine whether the entropy of the ambient image satisfies a threshold (906).

[0085] In response to computing device 100 determining the entropy satisfies a threshold, computing device 100 may obtain a first light source image and a second light source image (YES branch 906) (908). Computing device 100 may obtain the first light source image of subject 160 using camera 106 and a first light source of light sources 108. Computing device 100 may obtain the second light source image of subject 160 using camera 106 and a second light source of light sources 108. Computing device 100 may generate a ratio image (910). Computing device 100 may generate the ratio image based on the first light source image, the second light source image, and the ambient image.

[0086] In response to computing device 100 determining the entropy does not satisfy a threshold, computing device 100 may obtain a first light source image (NO branch 906) (912). Computing device 100 may obtain the first light source image of subject 160 using camera 106 and a first light source of light sources 108. Computing device 100 may generate a symmetry ratio image (914). Computing device 100 may generate the symmetry ratio image based on the first light source image.

[0087] Computing device 100 may determine a classification of subject 160 (916). Computing device 100 may determine the classification of subject 160 by at least providing the ratio image or symmetry ratio image to one or more machine learning models.Computing device 100 may perform an action based on the determined classification (918). For example, computing device 100 may selectively unlock access to a user interface or resources of computing device 100 based at least on the determined classification specifying subject 160 is a real user operating computing device 100. In another example, computing device 100 may output an identification of subject 160 based at least on the classification specifying an object type associated with the subject.

[0088] Example 1 : A method includes obtaining an ambient image with a camera; determining an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtaining, with the camera, a first image based on illumination from a first light source, obtaining, with the camera, a second image based on illumination from a second light source, generating a ratio image based on the first image, the second image, and the ambient image, and determining a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtaining,with the camera, the first image based on illumination from the first light source, generating a symmetry ratio image based on the first image, and determining the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and performing an action based at least on the classification.

[0089] Example 2: The method of example 1, wherein performing the action comprises: selectively unlocking access to a user device based at least on the classification specifying the subject is a real user operating the user device.

[0090] Example 3 : The method of any of examples 1 and 2, wherein performing the action comprises: outputting an identification of the subject based at least on the classification specifying an object type associated with the subject.

[0091] Example 4: The method of any of examples 1 through 3, wherein determining the entropy of the ambient image comprises: analyzing a distribution of pixel intensities of the ambient image; and determining, based on the distribution of pixel intensities, the entropy of the ambient image as an amount of information content or randomness included in the ambient image.

[0092] Example 5: The method of any of examples 1 through 4, wherein generating the ratio image comprises: determining a baseline corresponding to a baseline distance between the first light source and the second light source; determining a placement of the subject in a scene, wherein the scene is associated with an environment captured in the ambient image, and wherein the placement of the subject is defined by an angle specifying a location of the subject with respect to a vector perpendicular to a point on the subject; determining, based on the first image, a first distance between the first light source and the subject; determining, based on the second image, a second distance between the second light source and the subject; and generating the ratio image based at least on the baseline, the placement, the first distance, and the second distance.

[0093] Example 6: The method of any of examples 1 through 5, wherein generating the ratio image comprises: subtracting the first image from the ambient image; and subtracting the second image from the ambient image.

[0094] Example 7: The method of any of examples 1 through 6, wherein generating the ratio image comprises: providing the one or more machine learning models the ambient image, the first image, and the second image; generating, by the one or more machine learning models, an ambient intermediate representation based on the ambient image; generating, by the one or more machine learning models, a first intermediate representation based on the first image;generating, by the one or more machine learning models, a second intermediate representation based on the second image; and generating the ratio image based at least on the ambient intermediate representation, the first intermediate representation, and the second intermediate representation.

[0095] Example 8: The method of any of examples 1 through 7, wherein determining the classification of the subject comprises: determining a texture of the subject with a first machine learning model of the one or more machine learning models; determining a shape of the subject with a second machine learning model of the one or more machine learning models; determining a consistency of the first image and the second image with a third machine learning model of the one or more machine learning models; determining one or more facial features of the subject with a fourth machine learning model of the one or more machine learning models; determining an indicative area of a scene associated with the ambient image with a fifth machine learning model of the one or more machine learning models; determining one or more occlusions in the scene with a sixth machine learning model of the one or more machine learning models; and classifying the subject based on at least one of the texture, the shape, the consistency, the one or more facial features, the indicative area, and the one or more occlusions.

[0096] Example 9: The method of any of examples 1 through 8, further includes training the one or more machine learning models to classify the subject based at least on training data including ratio images of known subjects generated based on images taken by the camera.

[0097] Example 10: The method of any of examples 1 through 9, wherein the camera is a near infrared camera.

[0098] Example 11 : The method of any of examples 1 through 10, wherein the first light source is a first light emitting diode (LED) and the second light source is a second LED.

[0099] Example 12: A computing device includes a camera; a first light source; a second light source; at least one processor; and at least one non-transitory computer-readable storage medium storing instructions that are executable by the at least one processor to: obtain an ambient image with the camera; determine an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtain, with the camera, a first image based on illumination from the first light source, obtain, with the camera, a second image based on illumination from the second light source, generate a ratio image based on the first image, the second image, and the ambient image, and determine a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying thethreshold: obtain, with the camera, the first image based on illumination from the first light source, generate a symmetry ratio image based on the first image, and determine the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and perform an action based at least on the classification.

[0100] Example 13: The computing device of example 12, wherein to perform the action, the instructions are executable by the at least one processor to: selectively unlock access to the computing device based at least on the classification specifying the subject is a real user operating the computing device.

[0101] Example 14: The computing device of any of examples 12 and 13, wherein to perform the action, the at least one non-transitory computer-readable storage medium store instructions that are executable by the at least one processor to: output an identification of the subject based at least on the classification specifying an object type associated with the subject.

[0102] Example 15: The computing device of any of examples 12 through 14, wherein to determine the entropy of the ambient image, the instructions are executable by the at least one processor to: analyze distribution of pixel intensities of the ambient image; and determine, based on the distribution of pixel intensities, the entropy of the ambient image as an amount of information content or randomness included in the ambient image.

[0103] Example 16: The computing device of any of examples 12 through 15, wherein to generate the ratio image, the instructions are executable by the at least one processor to: determine a baseline corresponding to a baseline distance between the first light source and the second light source; determine a placement of the subject in a scene, wherein the scene is associated with an environment captured in the ambient image, and wherein the placement of the subject is defined by an angle specifying a location of the subject with respect to a vector perpendicular to a point on the subject; determine, based on the first image, a first distance between the first light source and the subject; determine, based on the second image, a second distance between the second light source and the subject; and generate the ratio image based at least on the baseline, the placement, the first distance, and the second distance.

[0104] Example 17: The computing device of any of examples 12 through 16, wherein to generate the ratio image, the instructions are executable by the at least one processor to: subtract the first image from the ambient image; and subtract the second image from the ambient image.

[0105] Example 18: The computing device of any of examples 12 through 17, wherein to generate the ratio image, the instructions are executable by the at least one processor to: generate, by the one or more machine learning models, an ambient intermediate representation based on the ambient image; generate, by the one or more machine learning models, a first intermediate representation based on the first image; generate, by the one or more machine learning models, a second intermediate representation based on the second image; and generate the ratio image based at least on the ambient intermediate representation, the first intermediate representation, and the second intermediate representation.

[0106] Example 19: The computing device of any of examples 12 through 18, wherein to determine the classification of the subject, the instructions are executable by the at least one processor to: determining a texture of the subject with a first machine learning model of the one or more machine learning models; determine a shape of the subject with a second machine learning model of the one or more machine learning models; determine a consistency of the first image and the second image with a third machine learning model of the one or more machine learning models; determine one or more facial features of the subject with a fourth machine learning model of the one or more machine learning models; determine an indicative area of a scene associated with the ambient image with a fifth machine learning model of the one or more machine learning models; determine one or more occlusions in the scene with a sixth machine learning model of the one or more machine learning models; and classify the subject based on at least one of: the texture, the shape, the consistency, the one or more facial features, the indicative area, and the one or more occlusions.

[0107] Example 20: The computing device of any of examples 12 through 19, wherein the at least one non-transitory computer-readable storage medium further store instructions that are executable by the at least one processor to: train the one or more machine learning models to classify the subject based at least on training data including ratio images of known subjects generated based on images taken by the camera.

[0108] Example 21 : The computing device of any of examples 12 through 20, wherein the camera is a near infrared camera.

[0109] Example 22: The computing device of any of examples 12 through 21, wherein the first light source is a first light emitting diode (LED) and the second light source is a second LED.

[0110] Example 23: A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device to: obtain an ambientimage with a camera; determine an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtain, with the camera, a first image based on illumination from a first light source, obtain, with the camera, a second image based on illumination from a second light source, generate a ratio image based on the first image, the second image, and the ambient image, and determine a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtain, with the camera, the first image based on illumination from the first light source, generate a symmetry ratio image based on the first image, and determine the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and perform an action based at least on the classification.

[0111] Example 24: The non-transitory computer-readable storage medium of example 23, wherein to perform the action, the instructions that, when executed, cause the at the least one processor of the computing device to: selectively unlock access to the computing device based at least on the classification specifying the subject is a real user operating the computing device.

[0112] Example 25: The non-transitory computer-readable storage medium of any of examples 23 and 24, wherein to perform the action, the instructions that, when executed, cause the at least one processor of the computing device to: output an identification of the subject based at least on the classification specifying an object type associated with the subject.

[0113] Example 26: The non-transitory computer-readable storage medium of any of examples 23 through 25, wherein to determine the entropy of the ambient image, the instructions that, when executed, cause the at least one processor of the computing device to: analyze distribution of pixel intensities of the ambient image; and determine, based on the distribution of pixel intensities, the entropy of the ambient image as an amount of information content or randomness included in the ambient image.

[0114] Example 27: The non-transitory computer-readable storage medium of any of examples 23 through 26, wherein to generate the ratio image, the instructions that, when executed, cause the at least one processor of the computing device to: determine a baseline corresponding to a baseline distance between the first light source and the second light source; determine a placement of the subject in a scene, wherein the scene is associated with an environment captured in the ambient image, and wherein the placement of the subject isdefined by an angle specifying a location of the subject with respect to a vector perpendicular to a point on the subject; determine, based on the first image, a first distance between the first light source and the subject; determine, based on the second image, a second distance between the second light source and the subject; and generate the ratio image based at least on the baseline, the placement, the first distance, and the second distance.

[0115] Example 28: The non-transitory computer-readable storage medium of any of examples 23 through 27, wherein to generate the ratio image, the instructions that, when executed, cause the at least one processor of the computing device to: subtract the first image from the ambient image; and subtract the second image from the ambient image.

[0116] Example 29: The non-transitory computer-readable storage medium of any of examples 23 through 28, wherein to generate the ratio image, the instructions that, when executed, cause the at least one processor of the computing device to: generate, by the one or more machine learning models, an ambient intermediate representation based on the ambient image; generate, by the one or more machine learning models, a first intermediate representation based on the first image; generate, by the one or more machine learning models, a second intermediate representation based on the second image; and generate the ratio image based at least on the ambient intermediate representation, the first intermediate representation, and the second intermediate representation.

[0117] Example 30: The non-transitory computer-readable storage medium of any of examples 23 through 29, wherein to determine the classification of the subject, the instructions that, when executed, cause the at least one processor of the computing device to: determining a texture of the subject with a first machine learning model of the one or more machine learning models; determine a shape of the subject with a second machine learning model of the one or more machine learning models; determine a consistency of the first image and the second image with a third machine learning model of the one or more machine learning models; determine one or more facial features of the subject with a fourth machine learning model of the one or more machine learning models; determine an indicative area of a scene associated with the ambient image with a fifth machine learning model of the one or more machine learning models; determine one or more occlusions in the scene with a sixth machine learning model of the one or more machine learning models; and classify the subject based on at least one of: the texture, the shape, the consistency, the one or more facial features, the indicative area, and the one or more occlusions.

[0118] Example 31 : The non-transitory computer-readable storage medium of any of examples 23 through 30, wherein the instructions that, when executed, further cause the atleast one processor of the computing device to: train the one or more machine learning models to classify the subject based at least on training data including ratio images of known subjects generated based on images taken by the camera.

[0119] Example 32: The non-transitory computer-readable storage medium of any of examples 23 through 31, wherein the camera is a near infrared camera.

[0120] Example 33: The non-transitory computer-readable storage medium of any of examples 23 through 32, wherein the first light source is a first light emitting diode (LED) and the second light source is a second LED.

[0121] Example 34: A computing system comprising means for performing any of the methods of examples 1-11.

[0122] Example 35: A computer program product for classifying a subject, the computer program product comprising at least one computer-readable storage medium encoded with instructions that cause one or more processors of a computing device to perform any of the methods of examples 1-11.

[0123] Example 36: Computer-readable storage medium encoded with instructions that cause one or more processors of a computing system to perform any of the methods of examples 1- 11.

[0124] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer- readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0125] By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can beused to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0126] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0127] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of intraoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0128] Various examples of the disclosure have been described. Any combination of the described systems, operations, or functions is contemplated. These and other examples are within the scope of the following claims.

Claims

CLAIMS:

1. A method comprising: obtaining an ambient image with a camera; determining an entropy of the ambient image; responsive to the entropy satisfying a threshold: obtaining, with the camera, a first image based on illumination from a first light source, obtaining, with the camera, a second image based on illumination from a second light source, generating a ratio image based on the first image, the second image, and the ambient image, and determining a classification from a plurality of classifications for a subject associated with the ambient image by at least providing the ratio image to one or more machine learning models; responsive to the entropy not satisfying the threshold: obtaining, with the camera, the first image based on illumination from the first light source, generating a symmetry ratio image based on the first image, and determining the classification for the subject associated with the ambient image by at least providing the symmetry ratio image to one or more machine learning models; and performing an action based at least on the classification.

2. The method of claim 1, wherein performing the action comprises: selectively unlocking access to a user device based at least on the classification specifying the subject is a real user operating the user device.

3. The method of any of claims 1 and 2, wherein performing the action comprises: outputting an identification of the subject based at least on the classification specifying an object type associated with the subject.

4. The method of any of claims 1 through 3, wherein determining the entropy of the ambient image comprises: analyzing a distribution of pixel intensities of the ambient image; and determining, based on the distribution of pixel intensities, the entropy of the ambient image as an amount of information content or randomness included in the ambient image.

5. The method of any of claims 1 through 4, wherein generating the ratio image comprises: determining a baseline corresponding to a baseline distance between the first light source and the second light source; determining a placement of the subject in a scene, wherein the scene is associated with an environment captured in the ambient image, and wherein the placement of the subject is defined by an angle specifying a location of the subject with respect to a vector perpendicular to a point on the subject; determining, based on the first image, a first distance between the first light source and the subject; determining, based on the second image, a second distance between the second light source and the subject; and generating the ratio image based at least on the baseline, the placement, the first distance, and the second distance.

6. The method of any of claims 1 through 5, wherein generating the ratio image comprises: subtracting the first image from the ambient image; and subtracting the second image from the ambient image.

7. The method of any of claims 1 through 6, wherein generating the ratio image comprises: providing the one or more machine learning models the ambient image, the first image, and the second image; generating, by the one or more machine learning models, an ambient intermediate representation based on the ambient image; generating, by the one or more machine learning models, a first intermediate representation based on the first image; generating, by the one or more machine learning models, a second intermediate representation based on the second image; and generating the ratio image based at least on the ambient intermediate representation, the first intermediate representation, and the second intermediate representation.

8. The method of any of claims 1 through 7, wherein determining the classification of the subject comprises: determining a texture of the subject with a first machine learning model of the one or more machine learning models; determining a shape of the subject with a second machine learning model of the one or more machine learning models; determining a consistency of the first image and the second image with a third machine learning model of the one or more machine learning models; determining one or more facial features of the subject with a fourth machine learning model of the one or more machine learning models; determining an indicative area of a scene associated with the ambient image with a fifth machine learning model of the one or more machine learning models; determining one or more occlusions in the scene with a sixth machine learning model of the one or more machine learning models; and classifying the subject based on at least one of: the texture, the shape, the consistency, the one or more facial features, the indicative area, and the one or more occlusions.

9. The method of any of claims 1 through 8, further comprising: training the one or more machine learning models to classify the subject based at least on training data including ratio images of known subjects generated based on images taken by the camera.

10. The method of any of claims 1 through 9, wherein the camera is a near infrared camera.

11. The method of any of claims 1 through 10, wherein the first light source is a first light emitting diode (LED) and the second light source is a second LED.

12. A computing device comprising means for performing any of the methods of claims 1-11.

13. A computing system comprising means for performing any of the methods of claims 1-11.

14. A computer program product for classifying a subject, the computer program product comprising at least one computer-readable storage medium encoded with instructions that cause one or more processors of a computing device to perform any of the methods of claims 1-11.

15. Computer-readable storage medium encoded with instructions that cause one or more processors of a computing system to perform any of the methods of claims 1-11.

Images