Gesture detection via subcutaneous tissue image capture from wrist pointing camera system

Abstract

Techniques for operating an AR system include determining a hand gesture formed by a user based on a sequence of two-dimensional images acquired from a near-infrared camera across a skin of a wrist of the user. Specifically, an image capture device disposed on a band worn on a user's wrist includes a source of electromagnetic radiation, such as a light emitting diode in the IR wavelength band in the user's wrist, and an infrared (IR) detector that produces a two-dimensional image sequence of a region within the dermis layer in the user's wrist. From this sequence, gesture detection circuitry determines a value of a biological flow metric, e.g., a change in a perfusion index (PI) between frames of the sequence, based on a trained model that generates the metric from the sequence. Finally, the gesture detection circuitry maps the value of the bio-flow metric to the specific hand / finger movement that determines the gesture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of and claims priority to U.S. application No. 17 / 809,772, filed on June 29, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present description relates to determining gestures for controlling objects in an augmented reality (AR) system. Background Art

[0004] Augmented reality (AR) is an interactive experience of a real-world environment in which objects that exist in the real world are enhanced by computer-generated sensory information, sometimes across multiple sensory modalities, including vision, hearing, and / or touch. Some AR systems provide such interactive experiences using smart glasses, which can provide visual information via cameras mounted on the frame and waveguides and lenses in the frame, and audio information and tactile information via the frame. Summary of the invention

[0005] The implementation described herein relates to identifying gesture-based inputs for an augmented reality (AR) system. For example, a user of an AR system may wish to pinch a virtual object (e.g., a pencil) to move it around in a display field. Therefore, a gesture detection system may need to recognize pinching motions. Some AR systems can distinguish fine differences in gesture motions. An example of this situation may be the distinction between an index finger-thumb pinch and a middle finger-thumb squeeze. Still further, different object selections may be indicated on a display with respect to different positions of the fingers. This gesture distinction may be difficult for conventional methods. However, an improved technique is based on the observation that changes in the perfusion index (PI) optically measured at the wrist can provide information about hand and finger movement. In addition, fully two-dimensional imaging of the PI can accurately pinpoint such changes sufficiently to make fine distinctions between gestures.

[0006] In one general aspect, a method may include capturing a sequence of images through the skin of a wrist of a user. The method may also include determining a biofluid flow metric based on the sequence of images. The method may further include determining a gesture formed by the user based on the biofluid flow metric. The method may further include triggering execution of a command related to an object being displayed in an augmented reality (AR) system based on the gesture.

[0007] In another general aspect, an augmented reality (AR) system includes an image capture device configured to capture a sequence of images across the skin of a wrist of a user. The AR system also includes a gesture detection circuit system coupled to a memory. The gesture detection circuit system is configured to determine a biofluid flow metric based on the sequence of images. The gesture detection circuit system is further configured to determine a gesture formed by the user based on the biofluid flow metric. The gesture detection circuit system is further configured to trigger execution of a command related to an object being displayed in the augmented reality (AR) system based on the gesture.

[0008] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit system, causes the processing circuit system to perform a method. The method may include capturing a sequence of images passing through the skin of a wrist of a user. The method may also include determining a biofluid flow metric based on the sequence of images. The method may further include determining a gesture formed by the user based on the biofluid flow metric. The method may further include triggering execution of a command related to an object being displayed in an augmented reality (AR) system based on the gesture.

[0009] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A An example system according to the implementations described herein is presented.

[0011] Figure 1B According to the implementation described in this article Figure 1A A front view of an example head mounted wearable device is shown, Figure 1C is a rear view of the head-mounted wearable device, and Figure 1D is a perspective view of the head-mounted wearable device.

[0012] Figure 2A is a diagram showing example illumination of capillaries in subcutaneous tissue with an infrared (IR) light emitting diode (LED).

[0013] Figure 2B is a diagram showing an example signal having an intensity depending on the density of red blood cells in a capillary.

[0014] Figure 3 is a diagram showing an example wrist-worn image capture device.

[0015] Figure 4is a diagram showing an example camera for detecting IR and / or RGB illumination in a channel pair within a wrist-worn image capture device.

[0016] Figure 5 is a diagram showing a bottom view of an example wrist-worn image capture device.

[0017] Figure 6 is a diagram showing a side view of an example wrist-worn image capture device with illumination provided by an IR LED.

[0018] Figure 7 is a diagram showing example gesture detection circuitry.

[0019] Figure 8 is a flow chart illustrating an example method according to the disclosed implementations.

[0020] Fig. 9 are diagrams showing examples of computer devices and mobile computer devices that can be used to implement the described techniques.

[0021] Fig.10 is a diagram showing an example of a distributed computer device that can be used to implement the described techniques. DETAILED DESCRIPTION

[0022] The challenge for AR systems is to provide a robust interface between the smart glasses 100 and the user. Some interfaces utilize user gestures to implement various commands. A robust interface may be able to distinguish between a thumb-index finger pinch and a thumb-middle finger pinch.

[0023] Conventional approaches to providing robust AR interfaces include using a world-facing RGB camera mounted on a smartglass frame to provide images from which skeletal hand tracking can be performed, thereby enabling full hand / finger interaction and gesture inference. However, there may be problems associated with using a world-facing RGB camera. For example, such a world-facing RGB camera mounted on a smartglass frame, typically near a hinge, has a camera sensor and an image stack processor (ISP) that consume a relatively large amount of power. Due to this large power consumption, a technical problem with using RBG images from a world-facing RGB camera to infer user gestures is that the world-facing RGB camera can only be used sparingly.

[0024] The technical solution for the above technical problem includes: determining the gesture formed by the user based on a sequence of images obtained from an infrared (i.e., near-infrared) camera through the skin of the user's wrist. Before discussing the details of this technical solution, Figures 1A to 1D See the example AR system in the .

[0025] Figure 1AA user is shown wearing an example head-mounted wearable device for use in an augmented reality (AR) system. In this example, for purposes of discussion and illustration, the example head-mounted wearable device 100 is in the form of example smart glasses that include display capabilities and computing / processing capabilities. The principles described herein can be applied to other types of eyewear with and without display capabilities and / or computing / processing capabilities. Figure 1B yes Figure 1A A front view of an example head mounted wearable device 100 is shown, Figure 1C is a rear view of the head-mounted wearable device, and Figure 1D is a perspective view of the head-mounted wearable device. As described above, in some examples, the example head-mounted wearable device 100 can take the form of a pair of smart glasses or augmented reality glasses.

[0026] like Figures 1B to 1D As shown, the example head-mounted wearable device 100 includes a frame 102. The frame 102 includes a front frame portion defined by a frame portion 103 and a bridge portion 109 connected to the frame portion 103, which surrounds a corresponding optical portion in the form of a lens 107. The arm portion 105 is coupled (e.g., pivotally or rotatably coupled) to the front frame via a hinge portion 110 at the corresponding frame portion 103. In some examples, the lens 107 can be a corrective / prescription lens. In some examples, the lens 107 can be an optical material that includes a glass and / or plastic portion that does not necessarily include a corrective / prescription parameter.

[0027] The display device 104 can be coupled to a portion of the frame 102. Figure 1B and Figure 1C In the example shown, the display device 104 is coupled in the arm portion 105 of the frame 102. With the display device 104 coupled in the arm portion 105, an eye box 140 extends toward the lens 107 for outputting content at an output coupler 144, where the content output by the display device 104 can be visible to the user. In some examples, the output coupler 144 can substantially coincide with the lens 107. In some examples, the head-mounted wearable device 100 can also include an audio output device 106 (such as, for example, one or more speakers), an illumination device 108, a sensing system 111, a control system 112, at least one processor 114, and an image capture device 116 or camera 116. The camera (or image capture device) 116 can capture an image via a shutter trigger or button; the shutter trigger is configured to cause a shutter to open to allow light reflected or scattered from the scene toward the image capture device 116 to be incident on a light detector (e.g., a charge coupled device (CCD) array, a photomultiplier tube, silver halide photographic film, etc.).

[0028] In some examples, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display source onto a portion of a teleprompter glass that acts as a beam splitter that sits at an angle (e.g., 30 to 45 degrees). The beam splitter may implement reflection and transmission values ​​that allow light from the display source to be partially reflected while the remaining light is transmitted. This optical design may allow a user to simultaneously see, for example, physical items in the world through the lens 107, alongside content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device 104. In some implementations, waveguide optics may be used to depict content on the display device 104.

[0029] In some examples, the head-mounted wearable device 100 may include a gaze tracking device 120, which includes, for example, one or more sensors 125 to detect and track eye gaze direction and movement. The data captured by the sensor 125 can be processed to detect and track gaze direction and movement as user input. In some examples, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices, including, for example, one or more processors 114 operably coupled to components of the control system 112. In some examples, the control system 112 may include a communication module that provides communication and information exchange between the head-mounted wearable device 100 and other external devices.

[0030] As described above, the challenge for AR systems is to provide a robust interface between the smart glasses 100 and the user. Some interfaces utilize user gestures to implement various commands. A robust interface may be able to distinguish between a thumb-index finger pinch and a thumb-middle finger pinch. For example, different gestures may indicate activation of different icons on the smart glasses display or indicate different objects to be moved within the display field.

[0031] Conventional approaches to providing robust AR interfaces involve using a world-facing RGB camera mounted on a smartglass frame to provide images from which skeletal hand tracking can be performed, enabling full hand / finger interaction and gesture inference. That is, an RGB camera mounted on the hinge of the frame will track hand and / or finger motions of the user's hand, and from this, infer gestures and implement commands associated with the gestures.

[0032] However, there may be problems associated with using a world-facing RGB camera. For example, such a world-facing RGB camera mounted on a smart glasses frame, typically near a hinge, has a camera sensor and an image stack processor (ISP) that consume a relatively large amount of power. Due to this large power consumption, a technical problem with using RBG images from a world-facing RGB camera to infer user gestures is that the world-facing RGB camera can only be used sparingly. For example, using a world-facing RGB camera, the onboard sensor can only detect gestures a few times a day based on the power consumption of the camera. In addition, the frame rate (which should be about 30 frames per second to obtain the kind of resolution used for fine hand movement distinction) may be less than 5 frames per second in operations with high latency.

[0033] Furthermore, even if the technical problem of high power consumption can be overcome, there is another technical problem involving possible occlusions within the narrow field of view; if the hand and / or fingers are not visible within the display field, gesture detection will not work.

[0034] Before continuing, a biological fluid flow metric is defined as a measurement of the flow of a biological fluid inside the body.Such a metric can be determined based on non-invasive optical measurements that produce a sequence of images.

[0035] There is some evidence that optical measurements of changes in the perfusion index (PI)—the ratio of pulsatile to non-pulsatile blood flow—contain useful information about hand and finger movements. This evidence is presented in FIG. 2A to FIG. 2B Readily available LED-photodiode (PD) pairs in commodity smartwatches can provide spatially sparse PI variation measurements that can be used to distinguish gestures that differ greatly from the interaction perspective, e.g., pinching versus waving, which are easily distinguishable. However, such sparse optical arrays are ill-equipped to provide the fine-scale differentiation sought here.

[0036] According to the implementation described herein, the technical solution to the above technical problem includes: determining a gesture formed by a user based on a sequence of images obtained from a near-infrared camera through the skin of the user's wrist. Specifically, an image capture device provided on a band worn on the user's wrist includes a source of electromagnetic radiation, for example, a light emitting diode (LED) that emits radiation into the user's wrist in an IR wavelength band (e.g., a narrow band of about 850 nm), and an infrared (IR) detector, and the IR detector generates a two-dimensional image sequence of an area within the dermis in the user's wrist. Through this sequence, a gesture detection circuit system determines a value of a biological fluid flow metric based on a training model that generates metrics from the sequence, for example, a change in a perfusion index (PI) between frames of the sequence. Finally, the gesture detection circuit system maps the value of the biological flow metric to a specific hand and / or finger movement that determines the gesture.

[0037] In some implementations, the training model includes a convolutional regression neural network that is trained based on a dataset that includes a two-dimensional image sequence and corresponding values ​​of a biological metric (e.g., PI changes).

[0038] In some implementations, the source of electromagnetic radiation (e.g., LED) is arranged on either side (e.g., left and right) of the detector. In some implementations, the detector forms a left channel and a right channel for radiation emitted from the source of electromagnetic radiation (LED). The two-dimensional image sequence then includes a left sequence and a right sequence from which the PI changes are determined.

[0039] In some implementations, the source of IR electromagnetic radiation is located on a camera that also includes a detector.

[0040] The technical advantage of this solution is that, compared to conventional methods, sufficiently low power is used so that the image capture device is always available and can provide high frame rates (e.g., 30 frames per second) and low latency (e.g., about 10 milliseconds). In addition, occlusion is no longer a problem because the illumination and detector are positioned a few millimeters away from the wrist. Specifically, the z-depth of the band on which the image capture device is mounted is controlled by the z-depth of the image capture device, rather than by the mounting height on the smart glasses frame.

[0041] Before describing an image capture device and a gesture detection circuit system configured to determine a gesture based on a sequence of two-dimensional images of the inside of a user's wrist, it will be helpful to explain the operating principles. FIG. 2A to FIG. 2B Explained in.

[0042] Figure 2A 2 is a diagram 200 showing an example illumination of capillaries 240 in subcutaneous tissue 236 with an infrared (IR) light emitting diode (LED) 210. Figure 2AAs shown, the IR LED emits radiation 212 in an IR wavelength band (e.g., a narrow band around about 850 nm). The radiation 212 is configured to pass through the skin surface 230, the stratum corneum 232, and the epidermis 234 without excessive brightness drop (i.e., the Beer-Lambert coefficient is sufficiently small).

[0043] like Figure 2A As shown, radiation 212 is incident on capillaries 240. Figure 2A As shown, capillaries 240 contain flowing blood. The flowing blood contains red blood cells 246, for example, hemoglobin molecules that can absorb radiation 212. Radiation 212 reflects from capillaries 240 to produce reflected radiation 242, which propagates through dermis 236, epidermis 234, stratum corneum 232, and skin surface 230 to IR detector 220, which then samples the signal generated by reflected radiation 242 at a specific sampling rate. For example, IR detector 220 is a photoplethysmography (PPG) sensor.

[0044] Figure 2B 2 is a diagram 250 showing an example signal 260 generated by the PPG sensor 220. Figure 2B As shown, the intensity of the signal 260 depends on the density of red blood cells in the capillaries 270, 280. Specifically, because red blood cells absorb IR radiation, capillaries 270 with a low red blood cell density produce signal peaks because less radiation 212 has been absorbed. Conversely, capillaries 280 with a high red blood cell density produce signal valleys because most of the radiation 212 has been absorbed.

[0045] In addition, there is further evidence that hand and finger movements induce different red blood cell densities in the capillaries. For example, a grasping motion can induce an increase in red blood cell density in the capillaries, which is seen as a drop in signal intensity at the PPG detector. This indicates that IR radiation signal intensity can indicate hand and / or finger movement, and such movement can define a gesture formed by a user of the AR system.

[0046] Furthermore, further analysis has demonstrated that this relationship between signal strength / radiation intensity and hand / finger movement can be captured using optical detectors in existing cameras, such as those found in the Intel® RealSense Depth Camera D435, which are repurposed for sensing on the wrist. The advantage of using such a camera rather than a small pixel detector such as a PPG detector is that the camera can generate a sequence of two-dimensional images of a larger field. Such images provide the resolution required for fine hand and finger movement differentiation.

[0047] Figure 33 is a diagram showing an example wrist-worn image capture device 300. The wrist-worn image capture device 300 includes an LED driver 310, an LED microcontroller 320, a mount 330, a camera 340, and an adjustable elastic wristband 350. Figure 3 As shown, wrist-worn image capture device 300 includes or is connected to gesture detection circuitry 360. This is merely one example of an image capture device and is not meant to be limiting.

[0048] The LED driver 310 is configured to provide current to the LED radiation source so that the brightness of the emitted radiation is relatively stable. This stability can be achieved by providing a constant current to the LED so that the intensity does not change if, for example, the power supply drops. It is important to keep the brightness stable in order to suppress noisy variations in the radiation intensity. Such variations may make it difficult to determine a biofluid flow metric (e.g., PI variations between frames). An example LED driver 310 is an Adafruit 12-channel 16-bit PWM LED driver from Adafruit Industries LLC.

[0049] The LED microcontroller 320 is configured to control the emission of LED radiation according to a particular pattern or schedule. For example, to enable generation of a two-dimensional image sequence, the LED microcontroller 320 may cause the LED radiation source to flash at a particular frequency (e.g., 30 times per second to produce a desired frame rate). The example LED microcontroller 320 uses an Adafruit QT Py 0 SAMD21 development board from Adafruit Industries.

[0050] The mount 330 is a 3D printed holder that is configured to hold the camera 340 in place within the image capture device 300. In some implementations, the mount 330 is attached to the camera 340 using two M3 screws. The mount 330 also includes a groove through which the wristband 350 can slide and lock into place. Again, this is just an example and should not be limiting.

[0051] The camera 340 is configured to receive reflected radiation from the inside of the user's wrist and generate frames or two-dimensional images over a particular field of view (e.g., 90 degrees by 60 degrees). In some implementations, the particular field of view is significantly larger than the angular range of the distance from the camera to the inside of the user's wrist in either dimension (e.g., x or y). An example camera 340 is an Intel® RealSense Depth Camera D435 manufactured by Intel Corporation. In some implementations, the images are raw images that are formatted for comparison between frames, for example, for absolute differences. In such implementations, the images can be normalized to prevent or minimize intensity values ​​from drifting over time. About Figure 4 Further details of camera 340 are shown.

[0052] Figure 4 is a diagram showing a camera 340 for detecting radiation in a channel pair within the wrist-worn image capture device 300. Figure 4 As shown, the camera 340 includes a right imager 410 , an IR projector 420 , a left imager 430 , and an RGB module 440 .

[0053] The left imager 430 and the right imager 410 are each configured to receive radiation reflected from the inside of the wrist and form a corresponding image on a detector. In some implementations, the left imager 430 and the right imager 410 each include an optic configured to focus the received radiation onto a corresponding detector. In some implementations, the optics and the detector are configured for IR radiation, for example, a narrow band around 850 nm. In some implementations, the left imager 430 and the right imager 410 (i.e., the left channel detector and the right channel detector) capture their corresponding images substantially simultaneously. This allows for more accurate gesture detection from both hands.

[0054] IR projector 420 is co-located in a single housing (i.e., camera housing) with left imager 430 and right imager 410, and is configured to emit IR radiation onto and into the user's wrist for reflection back. In some implementations, IR projection includes a laser that emits in IR (e.g., in a narrow band around 850 nm). In some implementations, IR projector 420 is configured to emit IR radiation into one of left imager 430 or right imager 410. In some implementations, IR projector 420 includes a beam splitter so that IR radiation is reflected back to both left projector 430 and right projector 410. In some implementations, IR projector 420 is soldered to the same circuit board as left projector 430 and right projector 410.

[0055] RGB module 440 is configured to generate and / or detect illumination in the visible spectrum. In some cases, radiation at the red end of the visible spectrum can have significant brightness (i.e., sufficiently small Beer-Lambert coefficient) inside the wrist. Therefore, RGB module 440 can provide an alternative imaging platform for generating frames for gesture detection.

[0056] It should be noted that the camera 340 described above is only an example and is not meant to be limiting. For example, the camera 340 need not include the RGB module 440.

[0057] Figure 5 is a diagram showing a bottom view 500 of the wrist-worn image capture device 300. Figure 5As shown, the wrist-worn image capture device 300 further includes a radiation barrier 510 and IR LED pairs 520 and 522 .

[0058] The radiation barrier 510 is configured to prevent radiation from leaking through the wrist-worn image capture device 300. In some implementations, the radiation barrier 510 is constructed of screened polyurethane foam for user comfort when wearing the wrist-worn image capture device 300. In some implementations, the radiation barrier 510 is approximately one quarter inch thick and is safe for use in contact with the user's skin.

[0059] IR LED pair 520 and 522 are configured to emit IR radiation into the inside of the user's wrist for reflection back into left imager 430 and right imager 410, respectively. An example of an IR LED used in wrist-worn image capture device 300 is a GaAlAs dual heterogeneous high-speed IR emitting diode centered at 850 nm manufactured by Vishay Semiconductors. It should be noted that the radiation can be provided by IR LED 520 and / or 522 and / or IR projector 420. Figure 6 Further details on the configuration of the IR LEDs are shown in .

[0060] Figure 6 is a diagram illustrating a planar side view of an example wrist-worn image capture device 300 with radiation provided by IR LED pairs 520 and 522. Figure 6 The illustrated diagram shows only LED pairs 520 and 522 and the respective radiation beams 612 and 616 that they emit.

[0061] Assume that Figure 6 In FIG. 5 , LED pairs 520 and 522 are symmetrically positioned relative to each other, so that the description of the configuration of LED pair 520 also applies to LED pair 522. Figure 6 As shown, the LEDs in LED pair 520 are aligned with housing 330 of wrist-worn image capture device 300. Thus, due to the geometry of housing 330, each LED in LED pair 520 has an axis of symmetry oriented at an angle 620 relative to a normal to the surface of radiation barrier 510. In some implementations, angle 620 is substantially equal to 35 degrees.

[0062] The radiation beam 612 has a half divergence angle 630, which is defined by the half intensity of the radiation emitted by the LED pair 520 relative to the intensity maximum emitted along the symmetry axis. In some implementations, the full divergence angle (i.e., twice the half divergence angle 630) is substantially equal to 22 degrees. Angles 620 and 630 are important in the placement of the IR detector relative to the LED pairs 520 and 522.

[0063] return Figure 3 And as previously mentioned, the wrist-worn image capture device 300 includes a gesture detection circuit system 360. The gesture detection circuit system 360 is configured to determine a biofluid flow metric (e.g., PI changes across frames) and, accordingly, determine a gesture based on the biofluid flow metric. Figure 7 Further details regarding gesture detection circuitry 360 are shown.

[0064] Figure 7 is a diagram illustrating an example gesture detection circuitry 360 connected to or embedded in the wrist-worn image capture device 300. The gesture detection circuitry 360 is configured to receive a two-dimensional image sequence from the camera 340 and determine a gesture formed by the user while generating the image sequence.

[0065] The gesture detection circuit system 360 includes a network interface 722, one or more processing units 724, and non-transitory memory 726. The network interface 722 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from the network into electronic form for use by the gesture detection circuit system 360. The set of processing units 724 includes one or more processing chips and / or assemblies. The memory 726 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid-state drives, etc. The set of processing units 724 and the memory 726 together form a control circuit system that is configured and arranged to perform various methods and functions as described herein.

[0066] In some implementations, one or more of the components of gesture detection circuitry 360 may be or may include a processor (e.g., processing unit 724) configured to process instructions stored in memory 726. Figure 7 As depicted, examples of such instructions include an image manager 730, a perfusion index model manager 740, a gesture model manager 750, and a gesture-based command manager 760. Further, as Figure 7 As shown, memory 726 is configured to store various data describing the respective managers that utilize such data.

[0067] The image manager 730 is configured to obtain image data 732 from the camera 340 of the wrist-worn image capture device 300. In some implementations, the image manager 730 obtains the image data 732 over a network via the network interface 722. In some implementations, the image manager 730 obtains the image data 732 through a direct connection. In some implementations, the image manager 730 obtains the image data 732 from a local storage device.

[0068] Image data 732 represents a two-dimensional sequence of images 732(1), 732(2), ... 732(N) captured by the wrist-worn image capture device 300. In some implementations, image data 732 represents a sample of images taken at a particular field inside the user's wrist. In some implementations, each of the sequence 732(1), 732(2), ... 732(N) is in the form of a heat map indicating the value of a biofluid flow metric, such as a perfusion index, at various key points within the particular field.

[0069] In some implementations, the sequence 732(1), 732(2)...732(N) captured by the wrist-worn image capture device 300 represents two-dimensional images captured at a specific duration (e.g., at a specific frame rate). In some implementations, the frame rate is 30 frames per second, i.e., the frames in the sequence 732(1), 732(2)...732(N) are captured at intervals of 1 / 30 second. Higher frame rates provide better temporal resolution for determining changes in biofluid flow metrics at key points. Additionally, if the sequence 732(1), 732(2)...732(N) is captured with low latency, gestures are accurately determined in real time. Figure 7 In FIG. 5 , the biological fluid flow rate is shown as a perfusion index (PI), or more specifically, the change in PI between frames.

[0070] In some implementations, sequences 732(1), 732(2), ... 732(N) include more than one independent channel, e.g., left and right channels from left and right detectors 430, 410 in camera 340. In such implementations, frames from different channels are modeled independently of each other.

[0071] The PI model manager 740 is configured to generate a PI model represented by PI model data 742 that maps the image data 732 to changes in PI over time. Figure 7 As shown, the PI model is a supervised model, specifically a convolutional regressor neural network, or a convolutional neural network with a regressor. In this context, the regressor is a set of key points for evaluating PI in a two-dimensional image sequence 732 (1), 732 (2) ... 732 (N). The model described herein is merely an example and should not be considered limiting. The PI model manager 740 includes a PI model training manager 741.

[0072] The PI model training manager 741 is configured to generate CNN regressor data 744 based on PI model training data 743. The PI model training data 743 includes a full set of two-dimensional image sequences 746 (1) ... 746 (T) and corresponding PI data 747 (1) ... 747 (T), wherein the corresponding PI data represents the PI value, or the change of the PI value between adjacent frames at the key point represented by the key point data 745. The PI model training manager 741 uses the PI model training data 743 and the Euclidean loss function to generate parameters of the hidden layer represented by the convolutional layer data 748, which has pooling layers and / or skip connections in some implementations. The PI model manager 740 then predicts the PI value at the final fully connected layer represented by the PI / FC layer data 749.

[0073] Gesture model manager 750 is configured to use gesture model data 752 (eg, delta PI data 754 representing PI changes between adjacent frames taken from FC layer data 749 ) to predict one or more gestures formed by a user while capturing image data 732 .

[0074] In some implementations, and as Figure 7 As shown, gesture model manager 750 predicts gestures based on deltaPI to gesture mapping data 756. In some implementations, mapping data 756 includes a lookup table that includes deltaPI values ​​and identifiers of hand and finger movements (e.g., "right grip," "left thumb-index finger pinch," etc.). In some implementations, the identifiers are numbers.

[0075] In some implementations, the gesture model manager 750 predicts gestures using a supervised neural network trained using PI values ​​and corresponding finger / hand motions. In some implementations, the finger / hand motions are represented as embeddings, i.e., low-dimensional vectors.

[0076] The gesture-based command manager 760 performs actions based on the gestures predicted by the gesture model manager 750. For example, a grasping motion may cause the gesture-based command manager 760 to search the AR display for objects within its field of view to grasp, and then move the object through the field of view according to the movement of the gesture.

[0077] The components (e.g., modules, processing unit 724) of gesture detection circuit system 360 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of gesture detection circuit system 360 can be configured to operate within a device cluster (e.g., a server cluster). In such implementations, the functionality and processing of the components of gesture detection circuit system 360 can be distributed to several devices of the device cluster.

[0078] The components of the gesture detection circuit system 360 may be or include any type of hardware and / or software configured to process attributes. Figure 7 One or more portions of the components shown in the components of the gesture detection circuit system 360 in FIG. 3 may be or may include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., computer code modules, a set of computer-readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of the components of the gesture detection circuit system 360 may be or may include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be included in conjunction with Figure 7 Where different modules and / or different components are shown, the functionality shown as two components may be combined into a single component.

[0079] Although not shown, in some implementations, the components (or portions thereof) of the gesture detection circuit system 360 may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, the components (or portions thereof) of the gesture detection circuit system 360 may be configured to operate within a network. Thus, the components (or portions thereof) of the gesture detection circuit system 360 may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN), a wide area network (WAN), etc. The network may be or may include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network may include one or more segments and / or may have portions based on various protocols such as the Internet Protocol (IP) and / or proprietary protocols. The network may include at least a portion of the Internet.

[0080] In some implementations, one or more of the components of the search system may be or may include a processor configured to process instructions stored in a memory. For example, image manager 730 (and / or a portion thereof), PI model manager 740 (and / or a portion thereof), gesture model manager 750 (and / or a portion thereof), and gesture-based command manager 760 (and / or a portion thereof) are examples of such instructions.

[0081] In some implementations, the memory 726 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, the memory 726 can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with components of the gesture detection circuit system 360. In some implementations, the memory 726 can be a database memory. In some implementations, the memory 726 can be or include non-local memory. For example, the memory 726 can be or include memory shared by multiple devices (not shown). In some implementations, the memory 726 can be associated with a server device (not shown) within a network and configured to serve the components of the gesture detection circuit system 360. Figure 7 As illustrated, memory 726 is configured to store various data, including image data 732 , PI model data 742 , and gesture model data 752 .

[0082] Figure 8 8 is a flow chart depicting an example method 800 for determining a gesture based on a sequence of two-dimensional images of the inside of a user's wrist. The method 800 may be combined with Figure 7 The gesture detection circuitry 360 is executed by the described software constructs, which reside in the memory 726 of the gesture detection circuitry 360 and are executed by the set of processing units 724.

[0083] At 802, image capture device 300 captures a sequence of images (e.g., two-dimensional image sequence 732(1), 732(2) ... 732(N)) across the skin of a user's wrist. For example, as the user forms a gesture, image capture device 300 causes a radiation source (e.g., IR LEDs 520, 522) to illuminate an area within the interior of the user's wrist, i.e., in the dermis where blood flows in capillary vessels. The radiation is reflected back toward an IR detector (e.g., detectors 430 and 410) that forms a two-dimensional image. The sequence is formed by LED microcontroller 420, which, for example, flashes the LED on and off at a particular frame rate.

[0084] At 804, the PI model manager 740 determines a biological fluid flow metric (e.g., a change or delta in PI between frames of the sequence) based on the image sequence 732(1), 732(2) ... 732(N). The model is a convolutional neural network with a regressor that takes keypoint data 745 representing keypoints of the regressor and predicts the change in PI between frames of the two-dimensional image sequence 732(1), 732(2) ... 732(N).

[0085] At 806, gesture model manager 750 determines a gesture formed by the user based on the biofluid flow metric (e.g., the change in PI between frames of two-dimensional image sequence 732(1), 732(2) ... 732(N)). This determination may be made using, for example, a lookup table or a supervised prediction model.

[0086] At 808 , the gesture-based command manager 760 triggers execution of a command related to an object being displayed in the AR system based on the gesture.

[0087] Fig. 9 An example of a general purpose computer device 900 and a general purpose mobile computer device 950 that can be used with the techniques described herein is shown. The computer device 900 is Figure 7 An example configuration of gesture detection circuit system 360 is shown.

[0088] like Fig. 9 As shown, computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only and are not intended to limit implementations of the present inventions described and / or claimed in this document.

[0089] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. Each of the components 902, 904, 906, 908, 910 and 912 is interconnected using various buses and can be installed on a common motherboard or installed in other ways where appropriate. The processor 902 can process instructions for execution within the computing device 900 (including instructions stored in the memory 904 or on the storage device 906) to display graphical information of a GUI on an external input / output device such as a display 916 coupled to the high-speed interface 908. In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memories where appropriate. Moreover, multiple computing devices 900 can be connected, each of which provides a portion of the necessary operations (for example, as a server row, a group of blade servers, or a multi-processor system).

[0090] The memory 904 stores information within the computing device 900. In one implementation, the memory 904 is one or more volatile memory units. In another implementation, the memory 904 is one or more non-volatile memory units. The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0091] The storage device 906 can provide mass storage for the computing device 900. In one implementation, the storage device 906 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a magnetic tape device, a flash memory or other similar solid-state memory device, or a device array, which includes devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 904, the storage device 906, or a memory on the processor 902.

[0092] The high-speed controller 908 manages bandwidth-intensive operations for the computing device 900, while the low-speed controller 912 manages less bandwidth-intensive operations. Such allocation of functions is merely an example. In one implementation, the high-speed controller 908 is coupled to a memory 904, a display 916 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 910 that can accept various expansion cards (not shown). In an implementation, the low-speed controller 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, through a network adapter.

[0093] As shown, computing device 900 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 920, or multiple implementations in such a server group. The computing device can also be implemented as a part of a rack server system 924. In addition, the computing device can be implemented in a personal computer such as a laptop computer 922. Alternatively, components from computing device 900 can be combined with other components in a mobile device (not shown) such as device 950. Each of such devices can contain one or more of computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 that communicate with each other.

[0094] The computing device 950 includes a processor 952, a memory 964, an input / output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The device 950 may also be provided with a storage device such as a micro drive or other device to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and several of these components may be mounted on a common motherboard or otherwise where appropriate.

[0095] The processor 952 may execute instructions within the computing device 950, including instructions stored in the memory 964. The processor may be implemented as a chipset including separate multiple analog and digital processor chips. For example, the processor may provide coordination of other components of the device 950, such as control of a user interface, applications executed by the device 950, and wireless communications performed by the device 950.

[0096] The processor 952 can communicate with the user through a control interface 958 and a display interface 956 coupled to a display 954. The display 954 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 956 may include suitable circuitry for driving the display 954 to present graphics and other information to the user. The control interface 958 can receive commands from the user and convert these commands for submission to the processor 952. In addition, an external interface 960 in communication with the processor 952 can be provided to enable the device 950 to communicate with other devices in the near area. The external interface 960 can, for example, provide wired communication in some implementations, or provide wireless communication in other implementations, and multiple interfaces can also be used.

[0097] The memory 964 stores information in the computing device 950. The memory 964 can be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 974 can also be provided, and the expansion memory can be connected to the device 950 through an expansion interface 972, which can include, for example, a SIMM (single in-line memory module) card interface. This expansion memory 974 can provide additional storage space for the device 950, or can also store applications or other information of the device 950. Specifically, the expansion memory 974 can include instructions for executing or supplementing the above-mentioned process, and can also include security information. Therefore, for example, the expansion memory 974 can be provided as a security module for the device 950, and can be programmed with instructions that allow the safe use of the device 950. In addition, security applications and additional information can also be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0098] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as a memory, expansion memory 974, or memory on processor 952, which may be received, for example, via transceiver 968 or external interface 960.

[0099] The device 950 can communicate wirelessly via a communication interface 966, which may include a digital signal processing circuit system when necessary. The communication interface 966 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, via a radio frequency transceiver 968. In addition, short-range communication can occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 970 can provide additional navigation and location-related wireless data to the device 950, which can be used by applications running on the device 950 where appropriate.

[0100] Device 950 may also communicate audibly using audio codec 960, which may receive verbal information from a user and convert it into usable digital information. Audio codec 960 may also generate audible sounds for the user, such as through a speaker in a handset of device 950, for example. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 950.

[0101] As shown, the computing device 950 can be implemented in a variety of different forms. For example, the computing device can be implemented as a cellular phone 980. The computing device can also be implemented as part of a smart phone 982, a personal digital assistant, or other similar mobile device.

[0102] Fig.10 Examples of devices that can be used with the techniques described herein include Figure 7 An example of a general purpose computer system 1000 of a gesture detection circuit system 360 is shown. Computing system 1000 is intended to represent various example forms of large-scale data processing devices, such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. Computing system 1000 may be a distributed system with multiple processors, possibly including network-attached storage nodes interconnected by one or more communication networks. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0103] The computing system 1000 may include any number of computing devices 1080a-d. The computing devices 1080a-d may include servers or rack servers, mainframes, etc. communicating via a local or wide area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.

[0104] In some implementations, each computing device may include multiple racks. For example, computing device 1080a includes multiple racks 1058a-1058n. Each rack may include one or more processors, such as processors 1052a-1052n and 1062a-1062n. The processors may include data processors, network attached storage devices, and other computer controlled devices. In some implementations, one processor may operate as a master processor and control scheduling and data distribution tasks. The processors may be interconnected through one or more rack switches 1062a-1062n, and one or more racks may be connected through switch 1078. Switch 1078 may handle communications between multiple connected computing systems 1000.

[0105] Each rack may include memory such as memory 1054 and memory 1064 and storage such as 1056 and 1066. Storage 1056 and 1066 may provide mass storage and may include volatile or non-volatile storage such as a network attached disk, a floppy disk, a hard disk, an optical disk, a tape, a flash memory, or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. Storage 1056 or 1066 may be shared between multiple processors, multiple racks, or multiple computing devices, and may include a computer-readable medium storing instructions executable by one or more processors. Memory 1054 and 1064 may include, for example, one or more volatile memory units, one or more non-volatile memory units, and / or other forms of computer-readable media such as magnetic or optical disks, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. Memory such as memory 1054 may also be shared between processors 1052a-1052n. Data structures such as indexes may be stored, for example, across storage 1056 and memory 1054. The computing system 1000 may include other components not shown, such as a controller, a bus, an input / output device, a communication module, etc.

[0106] The entire system may be composed of multiple computing devices 1000 that communicate with each other. For example, device 1080a may communicate with devices 1080b, 1080c, and 1080d, and these may be collectively referred to as computing devices 1000. As another example, Figure 7 The gesture detection circuit system 360 may include one or more computing devices 1000. Some of the computing devices may be located geographically close to each other, and other computing devices may be located geographically far away. The layout of the system 1000 is only an example, and the system may adopt other layouts or configurations.

[0107] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0108] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, device and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0110] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0111] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0112] A number of implementations have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0113] It should also be understood that when an element is referred to as being on another element, connected to another element, electrically connected to another element, coupled to another element, or electrically coupled to another element, the element may be directly on the other element, connected or coupled to the other element, or one or more intermediate elements may be present. On the contrary, when an element is referred to as being directly on another element, directly connected to another element, or directly coupled to another element, there is no intermediate element. Although the terms "directly on...", "directly connected to..." or "directly coupled to..." may not be used throughout the detailed description, the element shown as being directly on, directly connected to, or directly coupled to may be referred to as such. The claims of the present application may be modified to narrate the example relationships described in the specification or shown in the accompanying drawings.

[0114] Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the implementations. It should be understood that they are presented only by way of example and not limitation, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0115] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be excluded from the described processes, and other components can be added to or removed from the described systems. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A method comprising: capturing a sequence of images across the skin of a user's wrist; determining a biological fluid flow metric based on the sequence of images; determining a gesture formed by the user based on the biofluid flow metric; as well as Execution of a command related to an object being displayed in an augmented reality (AR) system is triggered based on the gesture.

2. The method of claim 1, wherein: Determining the gesture formed by the user includes: applying a first model that maps the image sequence to the biological fluid flow metric; and A second model is applied that maps the biological fluid flow metric to the gesture.

3. The method of claim 2, wherein: The first model includes a convolutional regression neural network; and Wherein, the method further comprises: The convolutional regression neural network is trained based on a data set including a sequence of images and corresponding values ​​of the biological fluid flow metric.

4. A method as claimed in any one of the preceding claims, wherein: The biometric comprises a change in a perfusion index between frames of the sequence of images.

5. A method as claimed in any one of the preceding claims, wherein: Capturing the sequence of two-dimensional images of the skin across the user's wrist comprises: emitting electromagnetic radiation from a radiation source into the interior of a wrist of the user; receiving the electromagnetic radiation reflected from the inside of the user's wrist into a radiation detector; and The sequence of images is formed by sampling the electromagnetic radiation reflected from the inside of the user's wrist into the radiation detector at a specified frame rate.

6. The method of claim 5, wherein: The radiation source comprises an infrared (IR) projector mounted in a camera, and the camera comprises the radiation detector.

7. The method according to claim 5 or 6, wherein: The radiation source comprises a left radiation source and a right radiation source; and Wherein, emitting the electromagnetic radiation from the radiation source to the inside of the wrist of the user comprises: Electromagnetic radiation is emitted from the left radiation source into a first portion of the interior of the user's wrist, and the electromagnetic radiation is emitted from the right radiation source into a second portion of the interior of the user's wrist.

8. The method of claim 7, wherein: Each of the left and right radiation sources includes a respective pair of infrared (IR) light emitting diodes (LEDs).

9. The method according to any one of claims 5 to 8, wherein: The radiation detector includes a left channel detector and a right channel detector; and Wherein, receiving the electromagnetic radiation reflected from the inside of the user's wrist into the radiation detector comprises: Electromagnetic radiation reflected from the inside of the user's wrist is received at the left channel detector and the right channel detector.

10. The method of claim 9, wherein: Sampling the electromagnetic radiation reflected from the inside of the user's wrist includes: A respective image is captured at each of the left channel detector and the right channel detector substantially simultaneously.

11. An augmented reality (AR) system, comprising: an image capture device configured to capture a sequence of images across the skin of a user's wrist; as well as a gesture detection circuitry coupled to the memory, the gesture detection circuitry configured to: determining a biological fluid flow metric based on the sequence of images; determining a gesture formed by the user based on the biofluid flow metric; as well as Execution of a command related to an object being displayed in an augmented reality (AR) system is triggered based on the gesture.

12. The AR system according to claim 11, wherein: The image capture device is disposed on a wristband worn on the user's wrist.

13. The AR system according to claim 11 or 12, wherein: The image capturing device comprises: a source of electromagnetic radiation, the source being configured to emit the electromagnetic radiation in an infrared (IR) wavelength band; and A detector is configured to detect the electromagnetic radiation reflected from an interior of the user's wrist, the detector being configured to detect electromagnetic radiation in the IR wavelength band.

14. The AR system of claim 13, further comprising: An IR controller is configured to control emission of the electromagnetic radiation by the source of the electromagnetic radiation according to a schedule.

15. The AR system according to claim 13 or 14, wherein: The source of electromagnetic radiation and the detector are co-located in a single housing.

16. The AR system according to any one of claims 13 to 15, wherein: The detector includes a left channel detector and a right channel detector; and Wherein, the source of electromagnetic radiation includes a first IR light emitting diode (LED) pair and a second IR LED pair, wherein the first IR LED pair is configured to emit the electromagnetic radiation so that the electromagnetic radiation is received at the left channel detector, and the second IR LED pair is configured to emit the electromagnetic radiation so that the electromagnetic radiation is received at the right channel detector.

17. A computer program product, the computer program product comprising a non-transitory storage medium, the computer program product comprising code, the code when executed by a processing circuit system causing the processing circuit system to perform a method, the method comprising: capturing a sequence of images across the skin of a user's wrist; determining a biological fluid flow metric based on the sequence of images; determining a gesture formed by the user based on the biofluid flow metric; as well as Execution of a command related to an object being displayed in an augmented reality (AR) system is triggered based on the gesture.

18. The computer program product of claim 17, wherein: Determining the gesture formed by the user includes: applying a first model that maps the two-dimensional image sequence to the biological fluid flow metric; and A second model is applied that maps the biological fluid flow metric to the gesture.

19. The computer program product of claim 18, wherein: The first model includes a convolutional regression neural network; and Wherein, the method further comprises: The convolutional regression neural network is trained based on a data set comprising a sequence of two-dimensional images and corresponding values ​​of the biological fluid flow metric.

20. The computer program product of any one of claims 17 to 19, wherein: The biometric comprises a change in a perfusion index between frames of the two-dimensional image sequence.

Images