Biometric detection using photodetector arrays

Abstract

A computing device, such as a wearable device, may include a light source and a photodetector array. The photodetector array may be used to determine touch events of a user occurring during a time interval. A subset of the photodetectors associated with the touch events provides a detection signal at each of a plurality of times within the time interval, and the detection signals may be aggregated to obtain a time series of aggregate detection signals. Biometric data of the user may be generated based on the time series of aggregate detection signals.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Non-Provisional Patent Application No. 17 / 305,990, entitled “Biometric Detection Using Photodetector Arrays,” filed July 19, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] Technical Field This text relates to biometric detection. [Background technology]

[0003] background There may be devices and techniques for capturing user biometrics. For example, various biometric monitoring techniques utilize light sources and photodetectors, such as light emitting diodes (LEDs) or photodiodes, at the user's skin surface to measure volumetric changes in the user's blood circulation. For example, photoplethysmography (PPG) sensors may be used to measure such volumetric changes to determine heart rate, respiration rate, and other biometric factors. Summary of the Invention

[0004] overview According to one general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium. The computer program product includes instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to determine a user touch event occurring at a photodetector array including a plurality of photodetectors during a time interval, and to determine a subset of the plurality of photodetectors associated with the touch event. The instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to aggregate detection signals from the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals, and to generate biometric data of the user based on the time series of aggregate detection signals.

[0005] According to another general aspect, a computer-implemented method includes determining a user touch event occurring at a photodetector array including a plurality of photodetectors during a time interval, and determining a subset of the plurality of photodetectors associated with the touch event, The method further includes aggregating detection signals from the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals, and generating biometric data of the user based on the time series of aggregate detection signals.

[0006] According to another general aspect, a computing device includes a processor, a storage medium storing instructions, a body, and a light source coupled to the body and configured to generate light in a direction of a user while the computing device is worn or held by a user. The computing device further includes a photodetector array including a plurality of photodetectors coupled to the body and configured to detect light of the light source reflected from the user. The instructions, when executed by the processor, cause the computing device to determine user touch events occurring at the photodetector array during a time interval from the light of the light source reflected from the user, determine a subset of the plurality of photodetectors associated with the touch events, aggregate detection signals from the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals, and generate biometric data of the user based on the time series of aggregate detection signals.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a system for biometric detection using a photodetector array. [Diagram 2] 2 is a flowchart showing an example of the operation of the system of FIG. 1. [Diagram 3] An example implementation of the system in Figure 1 is shown below. [Figure 4] Additional details regarding the example implementation of FIG. 3 are provided below. [Diagram 5] 4 illustrates an example motion detection technique that may be used in the implementation of FIG. 3. [Figure 6A] An example use case scenario for the example in Figure 5 is shown. [Figure 6B] An example use case scenario for the example in Figure 5 is shown. [Figure 6C] An example use case scenario for the example in Figure 5 is shown. [Figure 7A] 2 illustrates an example of a heart rate detection technique that may be implemented using the system of FIG. 1. [Figure 7B] 2 illustrates an example of a heart rate detection technique that may be implemented using the system of FIG. 1. [Figure 8] 2 is a block diagram showing an example implementation of touch event analysis in the system of FIG. 1. [Figure 9] FIG. 9 is a block diagram showing an example implementation of heart rate estimation in the system of FIG. 1 and the example of FIG. 8. [Figure 10] FIG. 2 is a block diagram of an example implementation of the system of FIG. 1 for authentication. [Figure 11] FIG. 2 is an exemplary diagram of the system of FIG. 1 implemented using a smartwatch. [Figure 12] FIG. 2 is an exemplary diagram of the system of FIG. 1 implemented using smart glasses. [Figure 13] FIG. 2 is an exemplary diagram of the system of FIG. 1 implemented using a smartphone. [Figure 14] 1 illustrates an example of a computing device, a mobile computing device, and a head-mounted device in accordance with at least one exemplary implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description The described systems and techniques enable biometric detection using photodetector arrays. Rather than using a single photodetector, e.g., a single photodiode, the described systems use multiple photodetectors and associated detection techniques to generate biometric data, such as heart rate, heart rate variability, or respiration rate.

[0010] By using multiple photodetectors in conjunction with the described detection techniques, various types of biometric data can be collected even when a user contacts the photodetector array in various ways or at any one of many different locations, and even when the user moves the contact point (e.g., the user's finger) relative to the photodetector array. Furthermore, biometric data can be collected while the user is simultaneously performing other functions using the photodetector array, such as controlling functions of a second device.

[0011] Some conventional devices perform biometric detection by requiring the user to touch a light detector or other type of detector in a predetermined manner. For example, a pulse oximeter may require the user to place a finger in a clip that contains a light source and a light detector. In other examples, a wearable device such as a fitness watch may be in contact with the user while being worn.

[0012] In these and other scenarios, as mentioned above, the PPG sensor may measure various biometric parameters of the user. Such conventional systems may perform threshold functions, for example, to ensure that the user's finger is placed in an expected position in a pulse oximeter relative to the light source / light detection being used, or to ensure that the PPG sensor of a wearable device (e.g., a watch) is sufficiently stable with respect to the user wearing it to ensure a desired level of accuracy of the PPG measurement.

[0013] While these and similar technologies offer many advantages, such technologies also have various drawbacks. For example, a user may be required to actively and accurately place the user's finger in a pulse oximeter for a minimal amount of time to obtain a desired reading. Similarly, some wearable devices, such as wearable watches, provide intermittent heart rate detection by requiring the user to touch a sensor coupled to the watch when a heart rate measurement is desired. In these and other examples, conventional PPG sensors in wearable devices add cost to such devices, may be inconvenient to use, and / or may not provide the desired type of reading or level of accuracy with respect to the captured biometric data.

[0014] For example, a PPG sensor used in a watch may detect volumetric changes in a user's blood flow in one or more of the user's arteries beneath the watch while the watch is being worn, thus making it possible to capture heart rate or other data without requiring the watch wearer to take any further specific action (such as touching the surface of a finger to the watch sensor).

[0015] However, in many cases, the user's arteries may move relative to the PPG sensor as part of normal user movement, and such movement may reduce the accuracy of the corresponding PPG measurement, especially when the user is moving and / or continuous measurements are required.

[0016] To address such difficulties, many conventional PPG sensors use additional hardware (and associated software) such as accelerometers and multiple light sources with multiple frequencies. For example, such devices may implement a multi-path approach that exploits the different characteristics of the different wavelengths of light used to account for arterial motion of the type just mentioned. For example, such a PPG sensor might use infrared (IR), red, and green light sources.

[0017] Such an approach increases the cost of the wearable device. Furthermore, different wavelengths of light have different levels of penetration into the user's skin and, accordingly, different abilities to detect and measure arterial changes in a desired manner. For example, IR light may be preferred in terms of achieving a desired level of skin penetration, but green light may be required to achieve the type of multipath solution mentioned above.

[0018] However, with the techniques described herein, it is possible to use a photodetector array of individual photodetectors to collect biometric data from any of a number of different subsets of the photodetector array during multiple touch events. Thus, biometric data may be collected across multiple touch events of a user and / or may account for the user's movements (including, for example, finger or arterial movements) during the measurement.

[0019] As a result, it is possible to perform passive, continuous heart rate measurements without having to use a multi-pass solution such as the multi-color approach mentioned above. The use of IR light in the described technique may, for example, provide deeper skin penetration, reduce cost, and increase reliability and accuracy.

[0020] Additionally, the desired measurements may be taken in a specified manner and for a specified time without requiring any affirmative action on the part of the user (such as requesting the user to touch a specified surface) for the purpose of taking a heart rate measurement. Instead, biometric measurements may be taken, for example, when the user touches the photodetector surface for some other purpose and / or without a specific or separate request to take a biometric measurement.

[0021] For example, as shown in the examples of Figures 3 and 4, an Optical Finger Navigation (OFN) surface of an OFN system may be configured to implement the techniques described herein to collect biometric data, such that designated biometric data may be passively and simultaneously collected when the OFN system is used for its intended purpose, such as controlling the functionality of a device, such as augmented reality (AR) glasses or other device.

[0022] For example, a user may use an OFN system repeatedly or frequently to control AR glasses or other devices. During such use, when the user contacts the OFN system in a manner determined to be compatible with biometric detection as described herein, such biometric detection may be performed over a large number of touch events performed by the user when controlling the AR glasses. By their nature, such touch events may be performed over a period of time, such as hours, days, or weeks. The user's heart rate data may then be captured, aggregated, and analyzed, for example, over the relevant period of time, thereby identifying trends or interpreting the collected data. Conclusions derived from such data analysis may be provided to the user, so that the user obtains the desired information in a cost-effective manner and in a manner that does not require a separate conscious action to initiate a heart rate measurement to obtain the desired information.

[0023] Figure 1 is a block diagram of a biometric detection system using a photodetector array, in which an array-based biometric detector 102 is shown worn or held by a user 104 as part of a computing device 106.

[0024] 1, the computing device 106 may be worn on the end of the arm of the user 104, such as on the user's wrist, hand, or finger. Thus, the computing device 106 may be implemented using, for example, a watch, band, strap, bracelet, or ring.

[0025] However, as will be noted in more detail below, such examples are non-limiting and the computing device 106 may be worn on any suitable or desired body part of the user 104 for purposes of capturing a desired gesture. For example, the computing device 106 may be worn on the user's leg (e.g., ankle) or may be incorporated into a head mounted device (HMD) including smart glasses. In yet other examples, the computing device 106 may be incorporated into clothing such as a glove, a sock, a belt, or a shirt sleeve. Specific examples of wearable versions of the computing device 106 of FIG. 1 are provided below with respect to FIGS. 11 and 12.

[0026] In additional or alternative examples, the computing device 106 may be held by the user 104. For example, FIG. 13, described below, illustrates an example of a held implementation of the computing device 106 using a smartphone. However, the computing device 106 may be configured as a held implementation in many different ways and using many different form factors. For example, the computing device 106 may be constructed as part of a joystick or other gaming device for gaming implementation, or as part of any held I / O device for controlling a computer or other device.

[0027] Thus, in general, wearing a computing device 106 refers to any scenario in which the computing device 106 remains on a body part of the user 104 without requiring further action on the part of the user 104. Holding a computing device 106 refers to a scenario in which the computing device 106 is maintained in contact with or in the physical possession of the user 104 as a result of actions of the user 104 maintained during holding. For example, such holding may include gripping, grasping, or carrying.

[0028] The computing device 106 can be both worn and held simultaneously or overlappingly. For example, the computing device 106 may be worn on one arm or hand and held in the other hand at the same time, or may be worn in a position where it can be held (e.g., at the end of a sleeve or on other clothing).

[0029] 1 further illustrates that computing device 106 may include a body 107. As will be understood from the various examples provided herein, body 107 may be any housing or casing suitable to allow mounting or holding of a particular implementation of computing device 106.

[0030] The body 107 may be configured to mount or couple to one or more sensors and other components, including a light source 108 and a light detector array 110, e.g., a light detector array. For example, the light source 108 may correspond to a light emitting diode (LED) that provides one or more wavelengths of light. The light detector array 110 may include a number of light detecting surfaces (e.g., pixels, or groups of pixels) configured to capture light from the light source 108 that is transmitted through or reflected from the user 104.

[0031] 1, the photodetector array 110 is also shown in an expanded view showing individual photodetectors of the photodetector array 110. For example, individual photodetectors of the photodetector array may correspond to individual pixels, groups of pixels, or regions of pixels.

[0032] In the simplified example of Figure 1, photodetector array 110 is shown as a 4 x 4 array. In various implementations, photodetector array 110 may be implemented in any desired size or shape, such as the n x n example of Figure 1 or n x m. Photodetector array 110 does not need to be periodic or symmetrical and may include any number of photodetectors grouped together for purposes of using the techniques described herein.

[0033] A close-up view of the photodetector array 110 includes photodetectors 110a, 110b, 110c, and 110d in a 4×4 array. The touch surface 105 corresponds to a detected point of contact between the user 104 and the photodetector array 110, which may occur during a touch event of the user 104 against the photodetector array 110.

[0034] For example, the touch surface 105 may correspond to the fingertip of a user 104 pressing against the surface of the photodetector array 110. In a more general example, the touch surface 105 may correspond to any skin surface of a user 104 in contact with the photodetector array 110. For example, an intervening surface (e.g., a screen protector) may be present between the skin surface of the user 104 and the photodetector array 110 if the intervening surface allows for sufficient levels of light transmission and reflection.

[0035] In a simplified example, the touch surface 105 may remain constant for a period of time. As will be described below, in other examples, the touch surface 105 may move in conjunction with the movement of the user 104.

[0036] 1, the touch surface 105 need only cover, contact, shield, or otherwise be associated with a portion (a subset of) of the individual photodetectors of the photodetector array 110. In this example, the touch surface 105 at least partially covers each of the photodetectors 110a, 110b, 110c, 110d.

[0037] As can also be observed, the coverage of the individual photodetectors 110a, 110b, 110c, 110d varies across the touch surface 105. For example, only a small portion of the individual photodetector 110a is covered by the touch surface 105, while almost the entirety of the individual photodetector 110b is covered.

[0038] Thus, some of the individual photodetectors 110a, 110b, 110c, 110d may be ineligible for use or inclusion by the array-based biometric detector 102 in determining biometric data of the user 104. For example, a signal detected by any of the individual photodetectors 110a, 110b may not provide useful biometric data due to low coverage by the touch surface 105.

[0039] Conversely, the signal detected by either of the individual photodetectors 110c, 110d may be used to determine biometric data, depending on a variety of other factors. However, even with the broad range of the individual photodetectors 110b, 110d shown, the detected signal from either or both of the individual photodetectors 110b, 110d may be discarded if the signal quality is poor, or if touch events are brief or sporadic, or if it is otherwise determined that the signal cannot be used to provide reliable biometric data.

[0040] Stated another way, the array-based biometric detector 102 may be configured to determine candidate photodetectors of the photodetector array 110 for potential use in determining biometric data in a touch event, and may further be configured to select a subset of the candidate photodetectors to generate the biometric data. A touch event may span a time interval during which the touch surface 105 may move, be repositioned, and / or remain stationary in conjunction with a corresponding movement of the user 104.

[0041] In some cases, for example, as mentioned above, touch events and corresponding touch surfaces and movements may result in unusable or unreliable data. Nevertheless, the array-based biometric detector 102 may collect a sufficient amount of data over a longer period of time (and corresponding number of touch events) and in a sufficient number and type of context to generate a significant and highly useful amount of biometric data. Furthermore, because the array-based biometric detector 102 is configured to determine whether and when to generate biometric data in conjunction with corresponding touch events, the user 104 does not need to take any specific action to generate the biometric data. Instead, the user 104 may use the photodetector array 110 for another purpose (e.g., optical finger navigation, etc., as described herein), and the array-based biometric detector 102 may passively collect the desired biometric data in a reliable, convenient, accurate, and cost-effective manner.

[0042] To perform the above and related functions, the array-based biometric detector 102 includes a frame buffer 120 configured to collect and store a series of image frames collected by the photodetector array 110. For example, the frame buffer 120 may continuously store image frames from the photodetector array 110 and may delete stored frames after a predetermined period of time, or after a maximum number of frames has been reached, or after a particular amount of memory has been used, and / or based on other deletion criteria. In other examples, during touch events occurring over a time interval, the frame buffer 120 may collect and store image frames from that time interval.

[0043] Touch event analyzer 122 may be configured to analyze frames from frame buffer 120. For example, touch event analyzer 122 may be configured to analyze a set of frames to determine whether occurrences of touch events captured in the set of frames are of sufficient quality and degree to be used to generate biometric data. For example, touch event analyzer 122 may detect candidate photodetectors having any degree of coverage by touch surface 105 and may perform a thresholding or gating function to select a usable subset of the candidate photodetectors for biometric data generation.

[0044] For example, touch event analyzer 122 may be configured to select for each frame a subset of photodetectors whose corresponding detection signals have sufficient quality to be used to generate biometric data, and may be further configured to determine whether the entire set of frames provides usable data for biometric data generation. In a more specific example, touch event analyzer 122 may be configured to perform spatio-temporal grouping of related pixel regions across multiple frames of frame buffer 120 over time, including measuring the consistency of the group set, to confirm that a usable touch event (e.g., a usable user finger touch) has occurred.

[0045] For example, the user 104 may touch the photodetector array 110 with varying degrees of firmness or smoothness. For example, the user 104 may move his / her finger (or other skin surface or contact point) in a smooth and continuous manner (as opposed to a sporadic, intermittent, or irregular manner) over the photodetector array 110. The touch event analyzer 122 may perform the type of spatiotemporal grouping mentioned above of detection signals associated with touch events, both in a single frame and across a series of frames, to evaluate the touch event as a whole.

[0046] If verified by touch event analyzer 122, the collected data for the touch events captured in a series of frames in frame buffer 120 may be processed and provided to biometric generator 124 to generate the desired type of biometric data. For example, a spectral analysis of a set of related frames may be performed for each frame (e.g., for a selected subset of photodetectors in each frame), such that measurements taken from the series of frames may provide time series data corresponding to biometric data of user 104, such as the heart rate of user 104.

[0047] As mentioned above, such biometric data may be processed and used for many purposes. For example, the biometric data may be analyzed in aggregate over a period of time to identify trends or unusual events. In another example, as described below with respect to FIG. 10, the biometric data may be used to authenticate a user 104 for use of a computing device 106 or another device.

[0048] As also noted above, biometric data may be collected while the user 104 is performing a separate function, such as optical finger navigation, using the computing device 106. Thus, the computing device 106 should be understood to include any computing device that may benefit from or is capable of performing the various functions just mentioned.

[0049] 1, computing device 106 is shown generally as including at least one processor 112, a non-transitory computer-readable storage medium 114, various types of input / output (I / O) hardware 116, and a function manager 118. Computing device 106a should be understood to potentially represent two or more computing devices in communication with each other, several examples of which are provided herein. Thus, at least one processor 112 may correspond to one or more processors on such a computing device that may be configured to execute instructions stored using non-transitory computer-readable storage medium 114, among other functions.

[0050] Because at least one computing device 106 may represent many types of computing devices, examples of which are provided herein, many types of input / output (I / O) hardware that may be used in such computing devices are illustrated collectively in Figure 1 as I / O hardware 116. By way of non-limiting example, such I / O hardware may include a display (e.g., a touch screen), buttons, speakers or audio sensors, tactile sensors or outputs, a camera, or peripherals (e.g., a mouse, keyboard, stylus).

[0051] In addition to the I / O hardware 116, at least one computing device 106a can execute and provide many types of software that utilize or are implemented by at least one processor 112, non-transitory computer-readable storage medium 114, and the I / O hardware 116. Such software (and related functionality) is included in FIG. 1 and is represented in FIG. 1 by function manager 118.

[0052] 1, function manager 118 should be understood to represent and encompass any software implementation that may be associated with the operation of I / O hardware 116. For example, function manager 118 may include an operating system and many different types of applications that may be provided by at least one computing device 106a, some examples of which are provided in more detail below.

[0053] For example, function manager 118 may be used to provide optical finger navigation using appropriate hardware from light source 108, photodetector array 110, and I / O hardware 116. Function manager 118 may also be configured to perform various types of biometric data analysis as described herein. Function manager 118 may also be configured to provide any supplemental functionality, such as authentication, that benefits from or can utilize the collected biometric data.

[0054] To facilitate operation of array-based biometric detector 102 by function manager 118, function interface 126 may be configured to communicate with function manager 118 to implement corresponding functions. As just mentioned, the corresponding functions may relate to any desired use or processing of the collected biometric data by function manager 118.

[0055] In some implementations, feedback characterizing the biometric data may be provided to the user 104. For example, the function interface 126 can instruct the function manager 118 to provide a visual, audio, or tactile indicator to the user 104 when the biometric data being collected indicates a negative event (e.g., an excessively high heart rate).

[0056] Figure 2 is a flow chart illustrating an example operation of the system of Figure 1. In the example of Figure 2, steps 202-208 are shown as separate sequential steps. However, in various example implementations, steps 202-208 may be performed in an overlapping or parallel manner, and / or in a nested, iterative, looped, or branched manner. Additionally, various steps or substeps may be included, omitted, or substituted.

[0057] 2, a user's touch event occurring at a photodetector array including multiple photodetectors during a time interval may be determined 202. For example, frames from a frame buffer 120 collected from the photodetector array 110 may be analyzed by a touch event analyzer 122 to determine whether a touch event has occurred.

[0058] A subset of the photodetectors associated with the touch event may be determined (204). For example, the touch event analyzer 122 may be configured to determine candidate photodetectors associated with each frame that may have sufficient signal strength and other characteristics for biometric data generation. The touch event analyzer 122 may test signals from the candidate photodetectors to select a subset of the photodetectors to be used for generating the biometric data. The subset of the photodetectors may be evaluated individually, as a whole, or on a frame-by-frame basis. Thus, frames may be evaluated individually or collectively. For example, group set consistency of the subset of photodetectors may be evaluated across multiple frames. Examples of touch event detection and analysis are provided in detail below, for example with respect to FIG. 8.

[0059] The detection signals from a subset of the multiple photodetectors at each of multiple times within the time interval may be aggregated to obtain a time series of aggregate detection signals 206. For example, the biometric generator 124 may perform spatial averaging and pooling of the signal strength of each frame such that the spatially averaged and pooled frames effectively form a time series.

[0060] Biometric data of the user may be generated based on the time series of the aggregate detected signals 208. For example, the biometric generator 124 may perform spectral analysis and filtering of the time series to obtain a heart rate measurement.

[0061] 3-6C illustrate an example use case scenario in which an optical finger navigation (OFN) module on a ring allows a user to navigate and interact with smart glasses (e.g., AR glasses) or other devices. The array-based biometric detector 102 of FIG. 1 may be configured to use such an OFN module on an AR ring to opportunistically monitor a user's biometrics, such as heart rate and heart rate variability measurements, without an explicit user trigger for a few seconds while the user 104 interacts with the AR glasses. As mentioned above, the array-based biometric detector 102 may be used to measure small vascular volume changes from a series of color-amplified OFN images. The array-based biometric detector provides an example of a passive health monitoring system that does not require cognitive load on the user 104 to initiate measurements.

[0062] Thus, although the described implementations may not be equipped with a dedicated PPG sensor, the described implementations can nevertheless decode optical imaging results from an OFN system and solve the blind source separation problem to accurately recover human pulse and other biometrics. As a result, for example, cardiovascular information can be stored daily and a screener can be built to alert the user of abnormal conditions (e.g., bradycardia, tachycardia) for early detection, even when the user 104 is simply performing normal daily activities.

[0063] Figure 3 illustrates an example implementation of the system of Figure 1 just mentioned above. In Figure 3, computing device 106 is implemented as a ring 302 having an optical finger navigation (OFN) surface 304. As mentioned above, OFN surface 304 utilizes photodetector array 110 and associated hardware / software (not shown separately in Figure 3) to enable thumb 306 of user 104 to provide input to or otherwise control functionality of ring 302 or another device. For example, user 104 may be wearing smart glasses and may use OFN surface 304 to control functionality of the smart glasses, such as screen navigation of a screen of the smart glasses.

[0064] Whenever the user 104 uses the OFN surface 304, the array-based biometric detector 102 can determine whether the current use of the OFN surface 304 constitutes a touch event that can be used to generate biometric data. For example, the surface of the thumb 306 can result in a touch surface that corresponds to the touch surface 105 of FIG. 1 as determined from reflected light associated with such touch surface. That is, reflected light associated with a touch surface (such as the touch surface 105 of FIG. 1) refers to the reflected light of light that is directed from a light source toward the user and reflected away from the user (e.g., from the surface of the user's thumb 306), where the reflected light is indicative of the touch surface in an image frame resulting from a touch event, as described herein.

[0065] For example, the ring 302 may be worn on the index finger of the user 104. The base of the thumb 306 may touch the portion of the ring 302 where the OFN surface 304 is visible.

[0066] Figure 4 shows additional details regarding the example implementation of Figure 3. Specifically, Figure 4 shows that an OFN module 402 can be used to provide the OFN surface 304 of Figure 3, and that the OFN module 402 can include an LED 404 and a photodetector array 406, which corresponds to the photodetector array 110 of Figure 1. As previously discussed, light from the LED 404 can be reflected off the thumb 306, and the reflected light can be detected by the photodetector array 406.

[0067] 5 illustrates a photodetector array 502 that provides a more detailed example of the photodetector array 406. A touch surface 504 may be detected by the OFN module 402. A left scroll 506 may be detected when a user moves the user's thumb 306 to the left. A right scroll 508 may be detected when a user moves the user's thumb 306 to the right.

[0068] Thus, using a mini LED light source (e.g., LED 404) that shines infrared light towards thumb 306 and a photodetector array 406 that receives the reflected light energy, the reflected light energy can be measured and used to control ring 302, AR glasses, or other devices.

[0069] Once contact between the thumb and the OFN is established, pixels in the OFN surface 304 (shown in FIG. 5 as an 8×8 photodetector array 502) can sense whether an area of ​​the finger is touching the photodetector array 502 by measuring reflected light energy. By aggregating the total response across multiple pixels to calculate optical flow, the OFN module 304 can determine the subtle direction of thumb movement. As mentioned, this direction estimate can be provided to the AR glasses interface to control menu screens or other operations of the AR glasses.

[0070] A user 104 may perform scrolling operations 506, 508, and many other types of navigation or use of the OFN module 402 at many different times throughout the day. The user 104 need not request detection or collection of biometric data in conjunction with such use of the OFN. Nevertheless, the array-based biometric detector 102 may be configured to determine whether it is possible to generate and collect biometric data for each potential touch event associated with use of the ring 302 of FIG.

[0071] 6A-6C provide example image frames illustrating touch events and associated context for spatiotemporal grouping strategies described herein. Such strategies can be used to determine whether a given time period is a suitable time for measuring a heart rate or generating biometric data.

[0072] For example, in Figure 6A, frame 602, frame 604, and frame 606 correspond to a series of frames captured and stored in frame buffer 120 of Figure 1 at times t = 0, t = 1, and t = N. In the example of Figure 6A, no touch event is detected.

[0073] 6B, frame 608 at time t=0 is shown as capturing touch surface 610. A subsequent frame 612 at time t=1 shows touch surface 616 positioned away from the previous touch surface 610 (represented by dotted line 614 in frame 612 for comparison). Similarly, a subsequent frame 618 at time t=N shows touch surface 622 positioned away from the previous touch surface 616 (represented by dotted line 620 in frame 618 for comparison).

[0074] 6B thus illustrates a case where a touch event has been made, but involves relatively fast and sporadic contact and movement of the touch surface. For example, FIG. 6B may represent use of the OFN module 402 to perform a click event, or may represent a non-thumb artifact, such as an accidental touch of the OFN surface 304. FIG. 6B thus illustrates a scenario where it is unlikely that a reliable heart rate measurement is possible or should be made, and thus may be rejected as a potential touch event for biometric data generation. For example, the touch event analyzer 122 may reject the example of FIG. 6B using techniques described in more detail below with respect to FIG. 8.

[0075] 6C, frame 624 at time t=0 is shown as capturing touch surface 626. A subsequent frame 629 at time t=1 shows touch surface 630 positioned away from the previous touch surface 626 (represented by dotted line 628 in frame 629 for comparison). Similarly, a subsequent frame 632 at time t=N shows touch surface 636 positioned away from the previous touch surface 630 (represented by dotted line 634 in frame 632 for comparison).

[0076] FIG. 6C thus illustrates a case where a touch event occurs and the touch event is spatially and temporally consistent and smooth. For example, FIG. 6C may represent or correspond to the example scrolling operation of FIG. 5. FIG. 6C thus illustrates a scenario where a reliable heart rate measurement is likely to occur and thus may be accepted as a potential touch event for biometric data generation. For example, touch event analyzer 122 may accept the example of FIG. 6C using techniques described in more detail below with respect to FIG. 8.

[0077] Figures 7A and 7B show examples of heart rate detection techniques implemented using the system of Figure 1. Figures 7A and 7B show that the system of Figure 1 can be used to mimic or provide a proxy or virtual PPG signal without requiring the use of a dedicated PPG sensor.

[0078] As mentioned above with respect to such PPG sensors, with each beat of the user 104, the user's heart pumps blood throughout the user's body. Thus, with respect to the thumb 306 of Figure 3, there is a small change in the volume of the blood vessels under the thumb 306. In general, light absorption from the light of the LED 404 of Figure 4 may depend on a variety of factors related to, for example, the user 104 and the light used.

[0079] For example, the reflected signal may be characterized by the degree of absorption by skin, bone, and tissue, and the degree of absorption by venous blood. Absorption by non-pulsatile arterial blood provides a substantially DC component in the reflected signal, while absorption by pulsatile arterial blood provides an AC signal. The AC signal is significantly smaller in magnitude than the DC component, but has an oscillation velocity that corresponds to the heart rate of the user 104. The resulting signal will be similar to the example of FIG. 7A showing a time-varying signal 702.

[0080] 1, as otherwise described herein, a signal 702 may be generated using multiple frames of the frame buffer 120. Then, as also shown in FIG. 7A, a mean-subtracted fast Fourier transform (FFT) of the waveform 702 may be calculated to represent the waveform 702 using phase-invariant features. An exemplary process may include first removing the mean components of the waveform 702 (e.g., originating from the various absorption parameters and associated DC components mentioned above) since the corresponding signal amplitude offsets do not describe information related to the heart rate of the user 104. An FFT signal 704 may then be calculated from the resulting zero-mean signal to obtain spectral features.

[0081] By obtaining such high amplitude frequencies, and because the human heart beats in a series of periodic events, a resultant frequency signal 706 can be obtained as shown in Figure 7B where the frequency signal 706 exhibits peaks 708 in a spectral display. The peaks can then be selected using maximum filtering techniques, as mentioned below, and then stored as an estimate of the human heart rate of the user 104.

[0082] The above example shows that pulse rate detection and collection of other biometric data can be performed in an opportunistic, all-day fashion, including the ability to decide whether to initiate relevant (such as heart rate) measurement algorithms at any given time. In the example, this decision can be made, for example, using the described techniques for spatiotemporally grouping related pixel regions over time and measuring the consistency of the group set to confirm a user's finger touch. An accurate pulse rate can then be calculated using the type of spectral analysis mentioned above once confidence is gained that a specified qualifying touch event has occurred.

[0083] Figure 8 is a block diagram illustrating an example implementation of touch event analysis in the system of Figure 1. That is, Figure 8 provides an example of a technique for determining whether to initiate generation and collection of pulse rate estimates or other biometric data.

[0084] In Figure 8, a set of buffered frames 802 may be obtained from frame buffer 120 of Figure 1. Each frame may include detection signals corresponding to touch surface 804, as described with respect to Figures 1, 5, and 6A-6C.

[0085] 8, the set of buffered frames 802 (e.g., a stream of images) may be provided to a contact binarization engine 806. The contact binarization engine 806 may be configured to convert each successive OFN image in the set of buffered frames 802 into a 0-1 image. The resulting binary image may be compressed to facilitate analysis. For example, the contact binarization engine 806 may generate the binary image using a threshold calibrated based on the degree to which an average thumb reflects light back to the OFN module 402.

[0086] The resulting stream of binary images then enters a convexity generator 808, which generates a three-dimensional characteristic of the convexity of the touch surface in the binary images. For example, the convex hull of each touch surface in each binary image can refer to a minimal convex set of shapes for each touch surface. For example, if the touch surfaces are considered as bounded subsets, the convex hull can be represented as the shape enclosed by the perimeter of the subset.

[0087] In relation to FIG. 8, the convexity generator 808 effectively characterizes the spatiotemporal smoothness of the touch surface within and between binary images to determine whether the touch surface represents a touch event suitable for biometric data generation, such as the type of touch event described and illustrated above with respect to FIG. 6C.

[0088] For example, such convexity properties can be determined using the formula: convexity_score=1 / ((convex hull of 3D binarized data)-(3D binarized data)). In this example, for perfectly convex 3D binarized data, convexity_score will be a high number. Thus, a convexity threshold can be set for convexity_score, above which the input stream of images can be verified as spatiotemporally coherent. If this threshold is met (yes), the set of buffered frames 802 can be provided to a heart rate estimation engine 810, which represents an example implementation of the biometric generator 124. If not (no), the process of FIG. 8 can be repeated for the next batch of buffered frames.

[0089] Figure 9 is a block diagram illustrating an example implementation for heart rate estimation in the system of Figure 1 and in the example of Figure 8. That is, Figure 9 illustrates an example implementation of the heart rate estimation engine 810 of Figure 8 that may be used once sufficient confidence is obtained from a spatiotemporal coherence check utilizing the spatial resolution of the OFN module 402 to enable an accurate measurement of the heart rate.

[0090] In Figure 9, a set of buffered frames 802 is shown as being input to a pooling engine 902. The pooling engine 902 performs a spatial average pooling of each image in the set of buffered frames 802. That is, from the binarization operation of Figure 8, a region of interest may be defined for each image, and the OFN values ​​within that image region may be averaged. In this way, each image may be summarized with a single number.

[0091] The resulting set of values ​​for the set of buffered frames acquired over a time interval gives a time series of values, thereby providing a signal similar to the PPG signal, as previously described with respect to Figures 7A and 7B. This time series can be provided to a spectrum analyzer 904, which can be configured to remove the time average of the time series signal and apply an FFT to represent high amplitude frequencies in the signal, as shown in Figure 7A. As already shown in Figure 7B, the human heart beats in a series of periodic events, resulting in a frequency signal of peaks in the spectral display. Maximum filtering can then be used to extract the value of the frequency signal, which corresponds to the determined estimate 906 of the heart rate of the user 104.

[0092] Figure 10 is a block diagram of an example implementation of the system of Figure 1 for authentication. Figure 10 shows that an array-based biometric detector 1002 may be configured to utilize a set of buffered frames 802 to generate a set of biometric data 1004 to be used by an authentication engine 1006 to authenticate a user 104 for use of the computing device 106 of Figure 1 or a related computing device. For example, the authentication engine 1006 may be configured to compare the set of biometric data 1004 with a pre-calibrated dictionary 1008 of stored biometric data of the user 104 to authenticate the user 104.

[0093] For example, during setup or initial use of the computing device 106, the user 104 may be requested and guided to participate in a calibration session designed to record particular gestures, navigation routines, or other uses of the photodetector array 110, such as tasks performed in using the OFN surface 304 of the OFN module 402. The calibration session may be designed to record any of the example sets of biometric data 1004, including finger size, finger shape, gesture patterns for particular gestures, morphology of the proxy PPG signal described above with respect to FIGS. 7A, 7B, and 9, and the resting heart rate of the user 104.

[0094] During subsequent use, the user can naturally interact with the computing device 106 (e.g., the ring 302 in FIG. 3). The associated identification parameters are provided by the array-based biometric detector 1002 and can be matched against a pre-calibrated dictionary 1008 by the authentication engine 1006.

[0095] An exemplary matching process may be performed using any suitable technique. For example, a classifier such as a decision tree may be used. Matching may assign different weights to individual data of the set of biometric data 1004. Matching may select individual data of the set of biometric data 1004 for use in a particular authentication process, such as by selecting the smallest set of parameters that have the highest quality measurements in the current interaction with the user 104.

[0096] Preferably, the user 104 does not need to initiate the authentication process. Instead, the user 104 can simply use the computing device 106, such as the ring 302, in a standard manner and authentication can occur in the background.

[0097] Fig. 11 is an example diagram of the system of Fig. 1 implemented using a smartwatch 1102. More specifically, Fig. 11 shows a bottom view of the smartwatch 1102, showing the light source 1104 and the photodetector array 1106. Thus, while worn by a user 104, the light source 1104 and the photodetector array 1106 can be appropriately positioned relative to the wrist of the user 104 to generate biometric data as described herein.

[0098] For example, the user 104 can perform normal daily activities while wearing the smartwatch 1102 normally. Furthermore, the smartwatch 1102 can be fastened by the user 104 with different levels of tightness. Thus, the touch surface defined between the user 104 (e.g., the user's wrist) and the smartwatch 1102 can also change. In other words, the smartwatch 1102 can move relative to the user 104 during normal wear.

[0099] However, using the techniques described herein, a changing / moving touch surface can be tracked and analyzed to provide reliable biometric data. For example, the touch surface can be tracked using optical flow techniques as described above in the context of optical finger navigation. For example, in a conventional smartwatch, without surface tracking and related concepts as described herein, even when multiple wavelengths of light are used, motion artifacts may be present that may confound the collection of desired data (e.g., heart rate measurement). However, as noted in FIG. 11 and above, it is possible to collect such biometric data using a single wavelength of light, e.g., IR light. Thus, the user 104 can enjoy passive biometric data detection that is reliably and accurately collected in a low-cost manner.

[0100] FIG. 12 is an example diagram of the system of FIG. 1 implemented using smart glasses 1202. As shown, the smart glasses 1202 may include a nose bridge module 1204 that includes a light source and a light detector array. The nose bridge module 1204 may move relative to the nose of the user 104, similar to the smart watch 1102 just described, but still function to capture buffer data using the techniques described herein. Additionally, while shown with a nose bridge attachment, similar techniques may be implemented in other locations on the smart glasses 1202, such as one or both arms of the smart glasses 1202. OFN modules may also be implemented in both the smart watch 1102 and the smart glasses 1202.

[0101] 13 is an example diagram of the system of FIG. 1 implemented using a smartphone 1302. As shown, the smartphone 1302 can include an OFN module 1304 implemented using the techniques described herein.

[0102] Although shown mounted on the side of the smartphone 1302, the OFN module 1304 may be mounted in any desired location on the smartphone 1302. For example, the OFN module 1304 may be mounted on the back of the smartphone 1302 opposite the screen of the smartphone 1302. The user 104 can then perform navigation functions using the OFN module 1304 while simultaneously viewing the screen of the smartphone 1302 and while biometric data is being passively collected.

[0103] FIG. 14 illustrates an example of a computing device 1400 and a mobile computing device 1450 that may be used with the techniques described herein. The computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, smart devices, home appliances, electronic sensor-based devices, televisions, servers, blade servers, mainframes, and other suitable computing devices. The computing device 1450 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementations described and / or claimed in this document.

[0104] The computing device 1400 includes a processor 1402, a memory 1404, a storage device 1406, a high-speed interface 1408 that connects to the memory 1404 and a high-speed expansion port 1410, and a low-speed interface 1412 that connects to a low-speed bus 1414 and the storage device 1406. The processor 1402 can be a semiconductor-based processor. The memory 1404 can be a semiconductor-based memory. Each of the components 1402, 1404, 1406, 1408, 1410, 1412 are interconnected using various buses and may be implemented on a common motherboard or in other manners as appropriate. The processor 1402 can process instructions for execution within the computing device 1400, including instructions stored in the memory 1404 or the storage device 1406, to display graphical information for a GUI on an external input / output device, such as a display 1416 coupled to the high-speed interface 1408. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Also, multiple computing devices 1400 may be connected, each providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

[0105] The memory 1404 stores information within the computing device 1400. In one implementation, the memory 1404 is one or more volatile memory units. In another implementation, the memory 1404 is one or more non-volatile memory units. The memory 1404 may be another form of computer-readable medium, such as a magnetic disk or optical disk. In general, the computer-readable medium may be a non-transitory computer-readable medium.

[0106] The storage device 1406 can provide mass storage for the computing device 1400. In one implementation, the storage device 1406 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods and / or computer-implemented methods as described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 1404, the storage device 1406, or a memory on the processor 1402.

[0107] The high-speed controller 1408 manages the bandwidth-intensive operations of the computing device 1400, while the low-speed controller 1412 manages the less bandwidth-intensive operations. Such an allocation of functions is merely an example. In one implementation, the high-speed controller 1408 is coupled to the memory 1404, the display 1416 (e.g., via a graphics processor or accelerator), and is coupled to a high-speed expansion port 1410 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 1412 is coupled to the storage device 1406 and the low-speed expansion port 1414. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0108] The computing device 1400 can be implemented in many different forms, as shown in the figure. For example, the computing device 1400 may be implemented as a standard server 1420, or multiple times within a group of such servers. The computing device 1400 can also be implemented as part of a rack server system 1424. Additionally, the computing device 1400 can be implemented in a computer, such as a laptop computer 1422. Alternatively, components from the computing device 1400 can be combined with other components in a mobile device (not shown), such as device 1450. Each such device can include one or more computing devices 1400, 1450, and the entire system can be made up of multiple computing devices 1400, 1450 that communicate with each other.

[0109] The computing device 1450 includes, among other components, a processor 1452, a memory 1464, an input / output device such as a display 1454, a communication interface 1466, and a transceiver 1468. The device 1450 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1450, 1452, 1464, 1454, 1466, 1468 are interconnected using various buses, and some of the components may be implemented on a common motherboard or in other manners as desired.

[0110] The processor 1452 can execute instructions within the computing device 1450, including instructions stored in the memory 1464. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide for coordination of other components of the device 1450, such as, for example, the user interface, applications run by the device 1450, and control of wireless communications by the device 1450.

[0111] The processor 1452 can communicate with a user via a control interface 1458 and a display interface 1456 coupled to a display 1454. The display 1454 can be, for example, a TFTLCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1456 can comprise suitable circuitry for driving the display 1454 to present graphical and other information to the user. The control interface 1458 can receive commands from the user and translate and submit them to the processor 1452. Additionally, an external interface 1462 can be provided in communication with the processor 1452 to enable short-range communication between the device 1450 and other devices. The external interface 1462 can, for example, provide wired communication in some implementations and wireless communication in other implementations, and multiple interfaces can be used.

[0112] The memory 1464 stores information within the computing device 1450. The memory 1464 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1484 is also provided, which may be connected to the device 1450 via an expansion interface 1482, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1484 may provide additional storage space to the device 1450 or may store applications or other information in the device 1450. In particular, the expansion memory 1484 may include instructions for performing or supplementing the aforementioned processes, and may also include secure information. Thus, for example, the expansion memory 1484 may be provided as a security module of the device 1450 and may be programmed with instructions that enable secure use of the device 1450. Additionally, secure applications may be provided via a SIMM card along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0113] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1464, expansion memory 1484, or memory on processor 1452, which may be received, for example, via transceiver 1468 or external interface 1462.

[0114] The device 1450 can wirelessly communicate via a communication interface 1466, which can include digital signal processing circuitry as needed. The communication interface 1466 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can be through, for example, a radio frequency transceiver 1468. Additionally, short-range communication can be provided, such as using Bluetooth, low-power Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 1480 can provide additional navigation and location related wireless data to the device 1450, which can be used as appropriate by applications executing on the device 1450.

[0115] Device 1450 can also communicate voice using audio codec 1460, which can receive spoken information from a user and convert it into usable digital information. Audio codec 1460 can also generate audible sounds for the user, such as through a speaker in a handset of device 1450. Such sounds can include sounds from a voice call, recorded sounds (e.g., voice messages, music files, etc.), and also sounds generated by applications running on device 1450.

[0116] The computing device 1450 may be implemented in many different forms, as shown in the figure. For example, the computing device 1450 may be implemented as a mobile phone 1483. The computing device 1450 may also be implemented as part of a smartphone 1481, a personal digital assistant, or other similar mobile device.

[0117] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, special purpose or general purpose, coupled to receive data and instruction formats and to transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0118] These computer programs (also known as modules, programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0119] To provide for user interaction, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or LED (light emitting diode)) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0121] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0122] In some implementations, the computing device shown in FIG. 14 can include sensors that interface with or are included in the HMD 1490. For example, one or more sensors included in the computing device 1450 or other computing devices shown in FIG. 14 can provide input to the HMD 1490 or generally provide input that can be used by the HMD 1490. The sensors include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, an ambient light sensor. The computing device 1450 (e.g., the HMD 1490) can use the sensors to determine an absolute position and / or a detected rotation of the HMD 1490 that can be used as input for use by the HMD 1490.

[0123] In some implementations, one or more input devices included in or connected to the computing device 1450 and / or the HMD 1490 may be used as inputs used by the HMD 1490. The input devices include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input capabilities, a game controller, or other connectable input devices.

[0124] In some implementations, one or more output devices included on the computing device 1450 and / or within the HMD 1490 can provide output and / or feedback to a user of the HMD 1490. The output and feedback can be visual, tactical, or audible. The output and / or feedback can include, but is not limited to, rendering a display of the HMD 1490, vibrating, turning one or more lights or strobes on and off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, playing an audio file. The output devices can include, but are not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe, a speaker.

[0125] In some implementations, the computing device 1450 can be placed within an HMD 1490 to create an integrated HMD system. The HMD 1490 can include one or more positioning elements that allow a computing device 1450, such as a smartphone 1481, to be placed in an appropriate position within the HMD 1490. In such implementations, the display of the smartphone 1481 can render images using the display of the HMD 1490.

[0126] In some implementations, the computing device 1450 may appear as another object in the computer-generated 3D environment. A user's interaction with the computing device 1450 (e.g., rotating, shaking, touching a touch screen, swiping a finger on a touch screen) can be interpreted as an interaction with an object in the AR / VR space. As just one example, the computing device may be a laser pointer. In such an example, the computing device 1450 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 1450, the user in the AR / VR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 1450 in the AR / VR environment on the computing device 1450 or on the HMD 1490.

[0127] In some implementations, the computing device 1450 may include a touch screen. For example, a user may interact with the touch screen in a particular manner that may mimic what is happening on the touch screen and on the display of the HMD 1490. For example, a user may use a pinch type of motion to zoom content displayed on the touch screen. This pinch type of motion on the touch screen may cause the information displayed on the display to zoom. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment.

[0128] In some implementations, one or more input devices (e.g., mouse, keyboard) in addition to the computing device can be rendered on the display of the HMD 1490. The rendered input devices (e.g., rendered mouse, rendered keyboard) can be used as rendered on the display.

[0129] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the description and claims.

[0130] Moreover, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve desirable results. In addition, other steps can be provided or removed from the described flows, and other components can be added to or removed from the described systems. Accordingly, other implementations are within the scope of the claims.

[0131] In addition to the above, controls are provided to the user that allow the user to choose whether and when the systems, programs, devices, networks, or features described herein enable collection of user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current location) and whether the user is sent content or communications from the server. Additionally, certain data may be processed in one or more ways before being stored or used such that the user information is deleted. For example, the user's identification information may be processed such that user information cannot be determined about that user, or the user's geographic location may be generalized where location information is obtained (e.g., to the city, zip code, state level, etc.) such that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

[0132] A computer system (e.g., a computing device) may be configured to wirelessly communicate with the network server over the network via a communications link established with the network server using any known wireless communication techniques and protocols, including radio frequency (RF), microwave frequency (MWF) and / or infrared frequency (IRF) wireless communication techniques and protocols suitable for communication over the network.

[0133] According to aspects of the disclosure, implementations of the various techniques described herein may be implemented in digital electronic circuitry, or computer hardware, firmware, software, or combinations thereof. Implementations may be implemented as a computer program product (e.g., an information carrier, a machine-readable storage device, a computer-readable medium, a computer program tangibly embodied in a tangible computer-readable medium) for processing by or controlling the operation of a data processing device (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that, when executed, cause a processor to perform a process. Computer programs such as the computer programs described above may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site, or may be distributed across multiple sites and interconnected by a communication network.

[0134] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations, however, implementations may be embodied in many alternative forms and should not be construed as being limited to only the implementations set forth herein.

[0135] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise. Furthermore, the terms "comprises," "comprising," "including," and / or "comprising," as used herein, are understood to specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0136] When an element is referred to as being "coupled," "connected," or "responsive to," or "on" another element, it is understood that the element can be directly coupled, connected, or responsive to, or on, the other element, or that intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive to," or "directly on," another element, there are no intervening elements present. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0137] For ease of description, spatially relative terms such as "below," "downward," "lower," "upward," "above," and the like may be used herein to describe an element or feature in relation to another element or feature depicted in the figures. It is understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in a figure is turned over, an element described as "below" or "below" the other element or feature would be oriented "above" the other element or feature. Thus, the term "downward" can encompass both an upward and downward orientation. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0138] Exemplary implementations of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized implementations (and intermediate structures) of the exemplary implementations. As such, variations in the shapes of the figures are expected as a result of manufacturing techniques, tolerances, and the like. Thus, exemplary implementations of the concepts described should not be construed as limited to the particular shapes of regions illustrated herein, but are intended to include deviations in shapes that result, for example, from manufacturing. Thus, the regions illustrated in the figures are schematic in nature, and the shapes are not intended to represent the actual shapes of regions of a device, and are not intended to limit the scope of the example implementations.

[0139] It is understood that, although terms such as "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. Thus, a "first" element can also be referred to as a "second" element without departing from the teachings of the present implementation.

[0140] Unless otherwise defined, the terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which these concepts belong. Furthermore, terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0141] While certain features of the described implementations have been shown as described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It should therefore be understood that the appended claims are intended to cover such modifications and changes that fall within the scope of the implementations. It should be understood that these are presented by way of example only, and not by way of limitation, and that various changes in form and details are possible. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different implementations described.

Claims

1. A method for implementing a method for a computer-implemented program comprising: determining user touch events occurring at a photodetector array including a plurality of photodetectors during a time interval; determining a subset of the plurality of photodetectors in which the touch event causes a detection signal from a photodetector of the subset of the plurality of photodetectors; aggregating the detection signals from the photodetectors of the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals; generating biometric data of the user based on the time series of aggregate detection signals; A computer program configured to:

2. 2. The computer program product of claim 1, wherein the instructions, when executed, are further configured to cause the at least one computing device to store a set of image frames captured by the photodetector array during the time interval, each image frame of the set of image frames including a touch surface of the touch event determined from reflected light of light directed from a light source toward the user and reflected from the user.

3. The instructions, when executed, cause the at least one computing device to: determining, for an image frame of the set of image frames, candidate photodetectors that receive reflected light associated with the touch surface of the image frame; selecting the subset of the plurality of photodetectors from the candidate photodetectors, including evaluating, for each of the candidate photodetectors, the reflected light associated with the touch surface against a threshold. The computer program product of claim 2 , further configured to:

4. The instructions, when executed, cause the at least one computing device to: performing, for each image frame of the set of image frames, a contact binarization on reflected light associated with a touch surface at each photodetector, thereby determining whether each photodetector is included in the subset of the plurality of photodetectors in a corresponding image frame; 4. A computer program according to claim 2 or claim 3, further configured to:

5. The instructions, when executed, cause the at least one computing device to: characterizing the spatiotemporal smoothness of the touch surface; 4. A computer program according to claim 2 or claim 3, further configured to:

6. The instructions, when executed, cause the at least one computing device to: characterizing a convexity of the touch surface for each image frame; determining, for each image frame, whether to include the detection signals from the photodetectors of the subset of photodetectors in the time series of aggregate detection signals based on the convexity; 4. A computer program according to claim 2 or claim 3, further configured to:

7. The instructions, when executed, cause the at least one computing device to: performing a spectral analysis on the aggregate detection signal for the set of image frames to determine frequency information in the aggregate detection signal; using the frequency information to generate the biometric data including a heart rate of the user; 4. A computer program according to claim 2 or claim 3, further configured to:

8. the instructions, when executed, are further configured to cause the at least one computing device to track a movement of a touch surface of the touch events during the time interval across the plurality of optical detectors, the touch surface including a contact point between the user and the plurality of optical detectors, the subset of the plurality of optical detectors including a photodetector from which the detection signal is obtained during the movement; A computer program according to claim 2 or claim 3.

9. The instructions, when executed, cause the at least one computing device to: determining an occurrence of the touch event from the photodetector array while the photodetector array is being used for optical finger navigation (OFN); 4. A computer program according to claim 2 or claim 3, further configured to:

10. The instructions, when executed, cause the at least one computing device to: performing a comparison of said biometric data with stored biometric data of said user; authenticating the user based on the comparison; 4. A computer program according to claim 2 or claim 3, further configured to:

11. determining user touch events occurring at a photodetector array including a plurality of photodetectors during a time interval; determining a subset of the plurality of photodetectors for which the touch event triggers a detection signal from a photodetector of the subset of the plurality of photodetectors; aggregating the detection signals from the photodetectors of the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals; generating biometric data of the user based on the time series of aggregate detection signals; A computer-implemented method comprising:

12. 12. The method of claim 11, further comprising storing a set of image frames captured by the photodetector array during the time interval, each image frame of the set of image frames including a touch surface of the touch event determined from reflected light of light directed from a light source toward the user and reflected from the user.

13. determining, for an image frame of the set of image frames, candidate photodetectors that receive reflected light associated with the touch surface of the image frame; selecting the subset of the plurality of photodetectors from the candidate photodetectors, including evaluating, for each of the candidate photodetectors, the reflected light associated with the touch surface against a threshold; The method of claim 12 further comprising:

14. characterizing a convexity of the touch surface for each image frame; determining, for each image frame, whether to include the detection signals from the photodetectors of the subset of photodetectors in the time series of aggregate detection signals based on the convexity; The method of claim 12 or claim 13, further comprising:

15. performing a spectral analysis on the aggregate detection signal for the set of image frames to determine frequency information in the aggregate detection signal; generating the biometric data including a heart rate of the user using the frequency information; The method of claim 12 or claim 13, further comprising:

16. 14. The method of claim 11, further comprising tracking a movement of a touch surface of the touch events during the time interval across the plurality of optical detectors, the touch surface including points of contact between the user and the plurality of optical detectors, the subset of the plurality of optical detectors including photodetectors from which detection signals are obtained during the movement.

17. The method of any one of claims 11 to 13, further comprising determining the occurrence of the touch event from the photodetector array while the photodetector array is being used for optical finger navigation (OFN).

18. 1. A computing device comprising: A processor; A storage medium for storing instructions; The main body, a light source coupled to the body and configured to generate light in a direction toward a user while the computing device is worn or held by the user; a photodetector array including a plurality of photodetectors coupled to the body and configured to detect light of the light source reflected from the user; Equipped with The instructions, when executed by the processor, cause the computing device to: determining touch events of the user occurring at the photodetector array during a time interval from the light of the light source reflected from a user; determining a subset of the plurality of photodetectors in which the touch event causes a detection signal from a photodetector of the subset of the plurality of photodetectors; aggregating the detection signals from the photodetectors of the subset of the plurality of photodetectors at each of a plurality of times within the time interval to obtain a time series of aggregate detection signals; generating biometric data of the user based on the time series of aggregate detection signals; Computing device.

19. The instructions, when executed by the processor, cause the computing device to: storing a set of image frames captured by the photodetector array during the time interval, each image frame of the set of image frames including a touch surface of the touch event; 20. The computing device of claim 18.

20. 20. The computing device of claim 18 or 19, incorporated into at least one of a smartwatch, a ring, smart glasses, or a smartphone.

Images