Gesture Detection by Image Capture of Subcutaneous Tissue from a Wrist Pointing Camera System

Abstract

Techniques for operating an AR system include determining hand gestures formed by a user based on a sequence of two-dimensional images through the skin of the user's wrist obtained from a near-infrared camera. Specifically, an image capture device disposed on a band worn around the user's wrist includes an electromagnetic radiation source, e.g., a light emitting diode (LED) in the infrared (IR) wavelength band that emits radiation within the user's wrist, and an IR detector that creates a sequence of two-dimensional images of regions within the skin layer of the user's wrist. From this sequence, a gesture detection circuit determines biometric flow metric values, e.g., changes in the perfusion index (PI) between frames of the sequence, based on a trained model that generates metrics from the sequence. Finally, the gesture detection circuit maps the biometric flow metric values to specific hand / finger movements that determine the gesture.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a continuation of U.S. application Ser. No. 17 / 809,772, filed Jun. 29, 2022, and claims the benefit of its priority. The disclosure of this U.S. application is hereby incorporated by reference in its entirety into this specification.

[0002] This description relates to determining gestures for controlling objects within an augmented reality (AR) system.

Background Art

[0003] Augmented reality (AR) is an interactive experience of a real - world environment where real - world objects are enhanced by computer - generated perceptual information, sometimes across multiple sensory modalities including vision, hearing, and / or touch. Some AR systems use smart glasses to provide such an interactive experience. Smart glasses can provide visual information via a camera mounted on a frame and waveguides and lenses within the frame, and can provide audio information and tactile information via the frame.

Summary of the Invention

[0004] The embodiments described herein relate to identifying gesture-based input for an augmented reality (AR) system. For example, a user of an AR system may wish to pinch a virtual object, such as a pen, to move the virtual object within a display field. Thus, a gesture detection system may need to identify the pinching gesture. Some AR systems may be able to distinguish fine differences in gesture actions. An example of this may be the distinction between pinching between the index finger and thumb and pinching between the middle finger and thumb. Taking this further, different positions of the fingers can indicate the selection of different objects on the display. Distinguishing such gestures can be difficult with conventional approaches. However, improved techniques are based on the finding that changes in the perfusion index (PI) optically measured at the wrist can provide information regarding hand and finger movements. Further, a full two-dimensional imaging of the PI can accurately identify such changes with sufficient accuracy to perform fine distinctions between gestures.

[0005] In a general aspect, a method can include capturing a sequence of images through the skin of a user's wrist. The method can also include determining a biofluid flow metric based on the sequence of images. The method can further include determining a gesture formed by the user based on the biofluid flow metric. The method can further include triggering the execution of a command related to an object displayed within an augmented reality (AR) system based on the gesture.

[0006] In another general aspect, an augmented reality (AR) system includes an image capture device configured to capture a sequence of images through the skin of a user's wrist. The AR system also includes a gesture detection circuit coupled to a memory. The gesture detection circuit is configured to determine a biofluid flow metric based on the sequence of images. The gesture detection circuit is also configured to determine a gesture formed by the user based on the biofluid flow metric. The gesture detection circuit is further configured to trigger execution of a command associated with an object displayed within the augmented reality (AR) system based on the gesture.

[0007] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product includes code that, when executed by a processing circuit, causes the processing circuit to perform a method. The method can include capturing a sequence of images through the skin of a user's wrist. The method can also include determining a biofluid flow metric based on the sequence of images. The method can further include determining a gesture formed by the user based on the biofluid flow metric. The method can further include triggering execution of a command associated with an object displayed within the augmented reality (AR) system based on the gesture.

[0008] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0009]

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Best Mode for Carrying Out the Invention

[0010] The challenge for an AR system is to provide a robust interface between the smart glasses 100 and the user. Some interfaces utilize the user's gestures to achieve various commands. A robust interface can distinguish between a thumb and index finger pinch and a thumb and middle finger pinch.

[0011] Conventional approaches to providing a robust AR interface involve using an RGB camera facing the world mounted on the smart glasses frame to provide images for full hand / finger interaction and gesture estimation for hand skeleton tracking. Nevertheless, there can be problems associated with the use of an RGB camera facing the world. For example, such an RGB camera facing the world mounted on the frame of the smart glasses, which is usually near the hinge, has a camera sensor that consumes a relatively large amount of power and an image signal processor (ISP). Due to this large power consumption, the technical problem associated with estimating the user's gestures using RGB images from the RGB camera facing the world is that the RGB camera facing the world can only be used sparingly.

[0012] A technical solution to the above technical problem involves determining the hand gestures formed by the user based on a sequence of images through the skin of the user's wrist obtained from an infrared (i.e., near-infrared) camera. Before explaining the details of this technical solution, an exemplary AR system is outlined in FIGS. 1A - 1D.

[0013] Figure 1A shows a user wearing an exemplary head-mounted wearable device for use in an augmented reality (AR) system. In this example, the exemplary head-mounted wearable device 100 takes the form of an exemplary pair of smart glasses for purposes of illustration and description, including a display function and a computing / processing function. The principles described herein may also apply to other types of eyewear with and without a display function and / or a computing / processing function. Figure 1B is a front view of the exemplary head-mounted wearable device 100 shown in Figure 1A. Figure 1C is a rear view of the exemplary head-mounted wearable device 100 shown in Figure 1A. Figure 1D is a perspective view of the exemplary head-mounted wearable device 100 shown in Figure 1A. As described above, in some examples, the exemplary head-mounted wearable device 100 may take the form of a pair of smart glasses or augmented reality glasses.

[0014] As shown in FIGS. 1B-1D, the exemplary head-mounted wearable device 100 includes a frame 102. The frame 102 includes a front frame defined by an edge portion 103 surrounding each optical portion in the form of a lens 107, and a bridge portion 109 connecting the edge portions 103. The arm portions 105 are each coupled to the front frame by a hinge portion 110 at the respective edge portion 103, e.g., pivotably or rotatably coupled. In some examples, the lens 107 may be a corrective lens / prescription lens. In some examples, the lens 107 may be an optical material including a glass portion and / or a plastic portion that does not necessarily incorporate corrective / prescription parameters.

[0015] The display device 104 can be coupled to a portion of the frame 102. In the examples shown in FIGS. 1B and 1C, the display device 104 is coupled to the arm portion 105 of the frame 102. With the display device 104 coupled to the arm portion 105, the eyepiece 140 extends towards the lens(es) 107 for output of content at the output coupler 144 where the content output by the display device 104 can be visible to the user. In some examples, the output coupler 144 may substantially coincide with the lens(es) 107. In some examples, the head-mounted wearable device 100 may also include an audio output device 106 (e.g., one or more speakers, etc.), an illumination device 108, a sensing system 111, a control system 112, at least one processor 114, an image capture device 116, or a camera 116. The camera (or image capture device) 116 can capture an image via a shutter trigger or button. The shutter trigger is configured to open the shutter and allow light reflected or scattered from the scene towards the image capture device 116 to be incident on a light detector (e.g., a charge-coupled device (CCD) array, a photomultiplier tube, a silver halide photographic film, etc.).

[0016] In some examples, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display light source onto a portion of a teleprompter glass that functions as a beam splitter mounted at an angle (e.g., 30 to 45 degrees). The beam splitter may allow reflection and transmission values that enable partial reflection of the light from the display light source while allowing the remaining light to be transmitted. Such an optical design may enable a user to view both physical items in the world, for example through the lens 107, adjacent to the content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device 104. In some embodiments, waveguide optics may be used to project content onto the display device 104.

[0017] In some examples, the head-mounted wearable device 100 may include a gaze tracking device 120 that includes, for example, one or more sensors 125 to detect and track the direction and movement of the user's line of sight. Data captured by the sensor(s) 125 may be processed to detect and track the direction and movement of the line of sight as user input. In some examples, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices that include, for example, one or more processors 114 operably coupled to components of the control system 112. In some examples, the control system 112 may include a communication module that provides for communication and exchange of information between the head-mounted wearable device 100 and other external devices.

[0018] The problem of the AR system described above is to provide a robust interface between the smart glasses 100 and the user. Some interfaces utilize the user's gestures to achieve various commands. A robust interface can distinguish between a thumb-index finger pinch and a thumb-middle finger pinch. For example, different gestures can include activating different icons on the smart glasses display or indicating different objects to be moved within the display field.

[0019] A conventional approach to providing a robust AR interface involves using a world-facing RGB camera mounted on the smart glasses frame to provide images for full hand / finger interaction and gesture estimation for hand skeleton tracking. That is, the RGB camera mounted on the frame hinge will track the hand movements and / or finger movements of the user's hand, thereby estimating gestures and achieving commands associated with the gestures.

[0020] Nevertheless, there may be problems associated with the use of a world-facing RGB camera. For example, such a world-facing RGB camera mounted on the frame of the smart glasses, which is usually near the hinge, has a camera sensor that consumes a relatively large amount of power and an image stack processor (ISP). Due to this large power consumption, the technical problem associated with estimating the user's gestures using the RGB images from the world-facing RGB camera is that the world-facing RGB camera can only be used sparingly. For example, when using a world-facing RGB camera, the on-board sensors can only detect gestures a few times a day based on the camera's power consumption. Furthermore, the frame rate (which needs to be about 30 frames per second to obtain a type of resolution for the difference in fine hand movements) can be less than 5 frames per second in operations with high latency.

[0021] Furthermore, even if the technical problem of high power consumption can be overcome, there are other technical problems including the possibility of occlusion within a narrow field of view. Gesture detection does not function when the hand and / or finger is not visible within the display field.

[0022] Before proceeding, biometric fluid flowmetry is defined as a measurement of the flow of a biological fluid within the body. Such a metric can be determined based on non-invasive optical measurements that create a sequence of images.

[0023] There is some evidence that optical measurements of changes in the pulsatility index (PI) (the ratio of pulsatile blood flow to non-pulsatile blood flow) contain useful information about hand and finger movements. This evidence is presented in FIGS. 2A-2B. A commercially available LED photodiode (PD) pair readily available within a smartwatch can provide spatially sparse PI change measurements that can be used to distinguish some gestures that can be broadly classified from an interaction perspective, such as a distinguishable pinch and waving a hand. Nevertheless, this sparse optical array is not adequately equipped to provide the fine differences required here.

[0024] According to the embodiments described herein, the technical solution to the above technical problem includes determining the gesture of the hand formed by the user based on a sequence of images through the skin of the user's wrist obtained from a near-infrared camera. Specifically, an image capture device disposed on a band worn around the user's wrist includes an electromagnetic radiation source, such as a light emitting diode (LED) in the infrared (IR) wavelength band (e.g., a narrow band near about 850 nm that emits radiation within the user's wrist), and an IR detector that creates a sequence of two-dimensional images of regions within the skin layer of the user's wrist. From this sequence, a gesture detection circuit determines a value of a biofluid flow metric, such as a change in the perfusion index (PI) between frames of the sequence, based on a trained model that generates a metric from the sequence. Finally, the gesture detection circuit maps the value of the biofluid flow metric to a specific hand and / or finger movement that determines the gesture.

[0025] In some embodiments, the trained model includes a convolutional regression neural network trained based on a dataset that includes a sequence of two-dimensional images and corresponding values of biological metrics such as PI changes.

[0026] In some embodiments, the electromagnetic radiation source (e.g., LED) is arranged on both sides (e.g., left and right) of the detector. In some embodiments, the detector forms a left channel and a right channel for the radiation emitted from the electromagnetic radiation source (LED). The sequence of two-dimensional images then includes a left sequence and a right sequence for which the change in PI is determined.

[0027] In some embodiments, the IR electromagnetic radiation source is located on a camera that also includes the detector. The technical advantage of the technical solution is that, in contrast to conventional approaches, the image capture device uses sufficiently low power so that it is always available and can provide a high frame rate (e.g., 30 frames per second) and low latency (e.g., about 10 milliseconds). Further, since the illumination and detector are positioned a few millimeters from the wrist, occlusion is no longer a problem. Specifically, the z-depth of the band on which the image capture device is mounted is controlled by the z-depth of the image capture device, for example, rather than the mount height on the smart glass frame.

[0028] Before describing the image capture device and gesture detection circuit configured to determine gestures based on a sequence of two-dimensional images inside the user's wrist, it is helpful to explain the principle of operation. This principle is illustrated in FIGS. 2A-2B.

[0029] FIG. 2A is a diagram 200 showing exemplary illumination of capillaries 240 within subcutaneous tissue 236 using an infrared (IR) light-emitting diode (LED) 210. As shown in FIG. 2A, the IR LED emits radiation 212 in the IR wavelength band (e.g., a narrow band around about 850 nm). The radiation 212 is configured to pass through the skin surface 230, stratum corneum 232, and epidermis 234 without excessive reduction in luminance. That is, the Lambert-Beer coefficient is sufficiently small.

[0030] As shown in FIG. 2A, the radiation 212 impinges on the capillaries 240. Also, as shown in FIG. 2A, the capillaries 240 contain flowing blood. The flowing blood contains red blood cells 246 such as hemoglobin molecules that can absorb the radiation 212, for example. The radiation 212 is reflected from the capillaries 240 to generate reflected radiation 242 that propagates through the dermis 236, epidermis 234, stratum corneum 232, and skin surface 230 to the IR detector, and the IR detector 220 then samples the signal created by the reflected radiation 242 at a specified sample rate. For example, the IR detector 220 is a photoplethysmography (PPG) sensor.

[0031] FIG. 2B is a diagram 250 showing an exemplary signal 260 generated by the PPG sensor 220. As shown in FIG. 2B, the intensity of the signal 260 depends on the red blood cell density in the capillaries 270, 280. Specifically, since red blood cells absorb IR radiation, the capillary 270 with a low density of red blood cells creates a signal peak because most of the radiation 212 is not absorbed. In contrast, the capillary 280 with a high density of red blood cells absorbs much of the radiation 212, creating a signal trough.

[0032] Furthermore, there is additional evidence that movement of the hand and fingers creates varying red blood cell densities within the capillaries. For example, a gripping motion can cause an increase in the red blood cell density within the capillaries, which is seen as a decrease in the signal intensity at the PPG detector. This indicates that the IR radiation signal intensity may indicate movement of the hand and / or fingers, and that such movement may define gestures formed by a user of the AR system.

[0033] Further analysis has demonstrated that such a relationship between signal intensity / radiation intensity and hand / finger movement can be captured using an off-the-shelf camera's optical detector, such as that seen in the Intel® RealSense Depth Camera D435, which is used for another purpose for sensing on the wrist. The advantage of using such a camera rather than a small pixel detector such as a PPG detector is that the camera can generate a sequence of two-dimensional images of a larger field. Such images provide the resolution required to distinguish fine hand and finger movements.

[0034] FIG. 3 is a diagram showing an exemplary wrist-worn image capture device 300. The wrist-worn image capture device 300 includes an LED driver 310, an LED microcontroller 320, a mount 330, a camera 340, and an adjustable elastic wristband 350. As shown in FIG. 3, the wrist-worn image capture device 300 includes or is connected to a gesture detection circuit 360. This is merely an example of an image capture device and is not intended to be limiting.

[0035] The LED driver 310 is configured to provide current to the LED radiation source such that the luminance of the emitted radiation is relatively stable. This stability can be achieved, for example, by providing a constant current to the LED so that the luminance does not change when the power supply drops. Maintaining stable luminance is important in order to suppress noisy fluctuations in the intensity of the radiation. Such fluctuations can make it difficult to determine biometric flow metrics (e.g., changes in PI between frames). An exemplary LED driver 310 is the Adafruit 12-channel 16-bit PWM LED driver from Adafruit Industries, LLC.

[0036] The LED microcontroller 320 is configured to control the emission of LED radiation according to a specified pattern or schedule. For example, to achieve the generation of a sequence of two-dimensional images, the LED microcontroller 320 can blink the LED radiation source at a specified frequency (e.g., 30 times per second to create a desired frame rate). An exemplary LED microcontroller 320 is the Adafruit QT Py 0 SAMD21 Dev Board from Adafruit Industries, LLC.

[0037] The mount 330 is a 3D printed holder configured to hold the gastric camera 340 in place within the image capture device 300. In some embodiments, the mount 330 is attached to the camera 340 using two M3 screws. The mount 330 also has a groove through which the wristband 350 can slide and lock in place. Again, this is merely an example and should not be limiting.

[0038] The camera 340 is configured to receive radiation reflected from inside the user's wrist and generate a two-dimensional image over a frame, or specified field of view (e.g., 90 degrees × 60 degrees). In some embodiments, the specified field of view is significantly larger in either dimension (e.g., x or y) than the angular range of the distance from the camera to the inside of the user's wrist. An exemplary camera 340 is the Intel® RealSense Depth Camera D435 by Intel Corp. In some embodiments, the image is an unprocessed image formatted for comparison between frames, e.g., for absolute difference. In such embodiments, the image can be normalized to prevent or minimize drift in luminance values over time. Further details of the camera 340 are shown with respect to FIG. 4.

[0039] FIG. 4 is a diagram showing the camera 340 for detecting radiation with a pair of channels within the wrist-worn image capture device 300. As shown in FIG. 4, the camera 340 includes a right image sensor 410, an IR projector 420, a left image sensor 430, and an RGB module 440.

[0040] The left imaging element 430 and the right imaging element 410 are each configured to receive radiation reflected from inside the wrist and form respective images on a detector. In some embodiments, the left imaging element 430 and the right imaging element 410 each include an optical system configured to focus the received radiation onto respective detectors. In some embodiments, the optical system and the detector are configured for IR radiation, such as a narrow band near 850 nm, for example. In some embodiments, the left imaging element 430 and the right imaging element 410 (i.e., the left channel detector and the right channel detector) capture their respective images substantially simultaneously. This enables more accurate gesture detection from both hands.

[0041] The IR projector 420 is disposed collocated with the left imaging element 430 and the right imaging element 410 within a single housing (i.e., the camera housing) and is configured to emit IR radiation onto and into the user's wrist for retroreflection. In some embodiments, the IR projection includes a laser that emits within the IR, such as a narrow band near 850 nm, for example. In some embodiments, the IR projector 420 is configured to emit IR radiation to one of the left imaging element 430 or the right imaging element 410. In some embodiments, the IR projector 420 includes a splitter such that the IR radiation is reflected back to both the left projector 430 and the right projector 410. In some embodiments, the IR projector 420 is soldered to the same circuit board as the left projector 430 and the right projector 410.

[0042] The RGB module 440 is configured to create and / or detect illumination within the visible spectrum. In some cases, radiation at the red end of the visible spectrum may have a significant luminance (i.e., a sufficiently small Lambert-Beer coefficient) inside the wrist. Thus, the RGB module 440 can provide an alternative imaging platform for generating frames for gesture detection.

[0043] It should be noted that the above camera 340 is only an example and is not intended to be limiting. For example, the camera 340 does not necessarily include the RGB module 440.

[0044] FIG. 5 is a diagram showing a bottom view 500 of the wrist-worn image capture device 300. As shown in FIG. 5, the wrist-worn image capture device 300 further includes a radiation baffle 510, and IR LED pairs 520 and 522.

[0045] The radiation baffle 510 is configured to block radiation leaking through the wrist-worn image capture device 300. In some embodiments, the radiation baffle 510 is constructed from a shift polyurethane foam for the comfort of the user while wearing the wrist-worn image capture device 300. In some embodiments, the radiation baffle 510 is 1 / 4 inch thick and is safe for use in contact with the user's skin.

[0046] The IR LED pairs 520 and 522 are each configured to emit IR radiation into the interior of the user's wrist for retroreflection into the left image sensor 430 and the right image sensor 410, respectively. An example of an IR LED used within the wrist-worn image capture device 300 is a GaAlAs Double Hetero high-speed IR light-emitting diode centered at 850 nm manufactured by Vishay Semiconductors. It should be noted that radiation can be provided by the IR LEDs 520 and / or 522, and / or the IR projector 420. Further details regarding the configuration of the IR LEDs are shown in FIG. 6.

[0047] FIG. 6 is a diagram showing a planar side view of an exemplary wrist-worn image capture device 300 with the radiation provided by the IR LED pairs 520 and 522. The diagram shown in FIG. 6 shows only the LED pairs 520 and 522 and their respective radiation beams 612 and 616 that they emit.

[0048] In FIG. 6, it is assumed that the LED pairs 520 and 522 are symmetrically placed with respect to each other such that the description of the configuration of LED pair 520 also applies to LED pair 522. As shown in FIG. 6, the LEDs within LED pair 520 are aligned with the housing 330 of the wrist-worn image capture device 300. Thus, due to the geometry of the housing 330, each LED of LED pair 520 has a symmetry axis that is oriented at an angle 620 with respect to the normal of the surface of the radiation baffle 510. In some embodiments, the angle 620 is substantially equal to 35 degrees.

[0049] The beam of radiation 612 has a divergence half-angle 630 that is defined by the intensity of the radiation emitted by the LED pair 520 being half the intensity of the maximum intensity emitted along the symmetry axis. In some embodiments, the divergence full angle, i.e., twice the divergence half-angle 630, is substantially equal to 22 degrees. The angles 620 and 630 are important in the placement of the IR detectors with respect to the LED pairs 520 and 522.

[0050] Returning to FIG. 3, as described above, the wrist-worn image capture device 300 includes a gesture detection circuit 360. The gesture detection circuit 360 is configured to determine biofluid flow metrics, such as PI changes across a frame, and thus determine gestures based on the biofluid flow metrics. Further details regarding the gesture detection circuit 360 are shown with respect to FIG. 7.

[0051] FIG. 7 is a diagram showing an exemplary gesture detection circuit 360 connected to or embedded within the wrist-worn image capture device 300. The gesture detection circuit 360 is configured to receive a sequence of two-dimensional images from the camera 340 and determine gestures formed by the user while the sequence of images is being generated.

[0052] The gesture detection circuit 360 includes a network interface 722, one or more processing units 724, and a non-transitory memory 726. The network interface 722 includes, for example, an Ethernet (registered trademark) adapter, a token ring adapter, etc. to convert an electronic signal and / or an optical signal received from a network into an electronic format for use by the gesture detection circuit 360. The set of processing units 724 includes one or more processing chips and / or processing assemblies. The memory 726 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid state drives, etc. The processing unit 724 and the memory 726 together form a control circuit configured and arranged to implement the various methods and functions described herein.

[0053] In some embodiments, one or more of the components of the gesture detection circuit 360 may be, or may include, a processor (e.g., the processing unit 724) configured to process instructions stored in the memory 726. Examples of such instructions shown in FIG. 7 include an image manager 730, a perfusion index model manager 740, a gesture model manager 750, and a gesture-based command manager 760. Further, as shown in FIG. 7, the memory 726 is configured to store various data, which will be described with respect to each manager that uses such data.

[0054] The image manager 730 is configured to obtain image data 732 from the camera 340 of the wrist-worn image capture device 300. In some embodiments, the image manager 730 obtains the image data 732 over a network via the network interface 722. In some embodiments, the image manager 730 obtains the image data 732 via a direct connection. In some embodiments, the image manager 730 obtains the image data 732 from a local storage device.

[0055] The image data 732 represents a sequence of two-dimensional images 732(1), 732(2),..., 732(N) captured by the wrist-worn image capture device 300. In some embodiments, the image data 732 represents samples of images captured on a specified field inside the user's wrist. In some embodiments, each of the sequences 732(1), 732(2),..., 732(N) takes the form of a heatmap indicating, for example, biometric fluid flow metric values such as perfusion index at various keypoints within the specified field.

[0056] In some embodiments, the sequences 732(1), 732(2),..., 732(N) captured by the wrist-worn image capture device 300 represent two-dimensional images captured over a specified period, for example, at a specified frame rate. In some embodiments, the frame rate is 30 frames per second. That is, the frames within the sequences 732(1), 732(2),..., 732(N) are acquired at intervals of 1 / 30 second. The higher the frame rate, the better the temporal resolution for determining changes in biometric fluid flow at the keypoints. Further, if the sequences 732(1), 732(2),..., 732(N) are captured with low latency, then, in turn, gestures are accurately determined in real time. In FIG. 7, the biometric fluid flow is shown as the perfusion index (PI), or more specifically, the change in PI between frames.

[0057] In some embodiments, the sequences 732(1), 732(2),..., 732(N) include, for example, a plurality of independent channels, such as a left channel and a right channel from the left detector 430 and the right detector 410 within the camera 340. In such embodiments, frames from different channels are modeled independently of each other.

[0058] The PI model manager 740 is configured to generate a PI model represented by PI model data 742 that maps image data 732 to the change in PI over time. As shown in FIG. 7, the PI model is a supervised model, specifically a convolutional regressor neural network, that is, a convolutional neural network having a regressor. In this context, the regressor is a set of key points within a sequence of two-dimensional images 732(1), 732(2),..., 732(N) where the PI is evaluated. The model described here is only an example and should not be construed as limiting. The PI model manager 740 includes a PI model training manager 741.

[0059] The PI model training manager 741 is configured to generate CNN regressor data 744 based on PI model training data 743. The PI model training data 743 includes a sequence of two-dimensional images 746(1),..., 746(T), and the corresponding PI data 747(1),..., 747(T) representing the change in PI, that is, the PI value between adjacent frames at the key points represented by the key point data 745. The PI model training manager 741 uses the PI model training data 743 and a Euclidean loss function to generate the parameters of the hidden layer represented by convolutional layer data 748 having a pooling layer and / or skip connections in some embodiments. The PI model manager 740 then predicts the PI value at the final fully connected layer represented by the PI / FC layer data 749.

[0060] The gesture model manager 750 is configured to predict one or more gestures formed by the user while the image data 732 was being captured using gesture model data 752, for example, differential PI data 754 representing the change in PI between adjacent frames obtained from the FC layer data 749.

[0061] In some embodiments, as shown in FIG. 7, the gesture model manager 750 predicts gestures based on the differential PI for the gesture mapping data 756. In some embodiments, the mapping data 756 includes, for example, a lookup table that includes values of differential PIs and identifiers of hand and finger movements, such as "right grip", "left thumb-index finger pinch", etc. In some embodiments, the identifiers are numbers.

[0062] In some embodiments, the gesture model manager 750 predicts gestures using a supervised neural network trained with PI values and corresponding finger / hand movements. In some embodiments, the finger / hand movements are represented as embeddings, i.e., low-dimensional vectors.

[0063] The gesture-based command manager 760 executes an action based on the gesture predicted by the gesture model manager 750. For example, a gripping action may cause the gesture-based command manager 760 to search the AR display for an object to grasp within the field of view and then move the object through the field of view according to the gesture movement.

[0064] The components of the gesture detection circuit 360 (e.g., modules, processing unit 724) can be configured to operate based on one or more platforms (e.g., one or more similar platforms or different platforms), and the one or more platforms can include one or more types of hardware, software, firmware, operating system, runtime library, etc. In some embodiments, the components of the gesture detection circuit 360 can be configured to operate within a cluster of devices (e.g., a server farm). In such embodiments, the functions and processing of the components of the gesture detection circuit 360 can be distributed across several devices of the device cluster.

[0065] The components of the gesture detection circuit 360 may be any type of hardware and / or software configured to process attributes, or may include them. In some embodiments, one or more portions of the components shown in the gesture detection circuit 360 of FIG. 7 may be hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memories), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions executable by a computer), or may include them. For example, in some embodiments, one or more portions of the components of the gesture detection circuit 360 may be software modules configured for execution by at least one processor (not shown), or may include it. In some embodiments, the functionality of the components may be included in different modules and / or different components other than those shown in FIG. 7, including combining the functionality shown as two components into a single component.

[0066] Although not shown, in some embodiments, components (or portions thereof) of the gesture detection circuit 360 can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some embodiments, components (or portions thereof) of the gesture detection circuit 360 can be configured to operate within a network. Thus, components (or portions thereof) of the gesture detection circuit 360 can be configured to function within various types of network environments that can include one or more devices and / or one or more server devices. For example, the network may be, or may include, a local area network (LAN), a wide area network (WAN), etc. The network may be, or may include, a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network can include one or more segments and / or can have portions based on various protocols such as the Internet Protocol (IP) and / or proprietary protocols. The network can include at least a portion of the Internet.

[0067] In some embodiments, one or more of the components of the search system can be, or can include, a processor configured to process instructions stored in a memory. For example, the image manager 730 (and / or a portion thereof), the PI model manager 740 (and / or a portion thereof), the gesture model manager 750 (and / or a portion thereof), and the gesture-based command manager 760 (and / or a portion thereof) are examples of such instructions.

[0068] In some embodiments, memory 726 may be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some embodiments, memory 726 may be implemented as a plurality of memory components (e.g., a plurality of RAM components or disk drive memory) associated with components of gesture detection circuit 360. In some implementations, memory 726 may be a database memory. In some implementations, memory 726 may be non-local memory or may include it. For example, memory 726 may be or may include memory shared by a plurality of devices (not shown). In some embodiments, memory 726 may be associated with a server device (not shown) in a network and configured to serve components of gesture detection circuit 360. As shown in FIG. 7, memory 726 is configured to store various data including image data 732, PI model data 742, and gesture model data 752.

[0069] FIG. 8 is a flowchart showing an exemplary method 800 for determining a gesture based on a sequence of two-dimensional images inside a user's wrist. Method 800 may be executed by a software construct described in connection with FIG. 7, which exists in memory 726 of gesture detection circuit 360 and is executed by a set of processing units 724.

[0070] At 802, the image capture device 300 captures a sequence of images (e.g., a sequence of two-dimensional images 732(1), 732(2),..., 732(N)) through the skin of the user's wrist. For example, when a gesture is formed by the user, the image capture device 300 causes a radiation source (e.g., IR LEDs 520, 522) to illuminate an area inside the user's wrist, i.e., an area within the dermis layer where blood flows within the capillaries. The radiation is retroreflected towards IR detectors (e.g., detectors 430 and 410) that form two-dimensional images. The sequence is formed, for example, by an LED microcontroller 420 that turns the LEDs on and off at a specified frame rate.

[0071] At 804, the PI model manager 740 determines biofluid flow metrics (e.g., change in PI or differential PI between frames of the sequence) based on the sequence of images 732(1), 732(2),..., 732(N). The model is a convolutional neural network with a regressor that captures keypoint data 745 representing keypoints of the regressor and predicts the change in PI between frames of the sequence of two-dimensional images 732(1), 732(2),..., 732(N).

[0072] At 806, the gesture model manager 750 determines the gesture formed by the user based on biofluid flow metrics, e.g., the change in PI between frames of the sequence of two-dimensional images 732(1), 732(2),..., 732(N). This determination can be made, for example, using a lookup table or a supervised prediction model.

[0073] At 808, the gesture-based command manager 760 triggers the execution of commands related to objects displayed within the AR system based on the gesture.

[0074] FIG. 9 is a diagram showing an example of a general computer device 900 and a general mobile computer device 950 that can be used with the technology described herein. The computer device 900 is an exemplary configuration of one of the gesture detection circuits 360 of FIG. 7.

[0075] As shown in FIG. 9, the computing device 900 is intended to represent various forms of digital computers such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing device 950 is intended to represent various forms of mobile devices such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0076] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and the high-speed expansion port 910, and a low-speed interface 912 connected to the low-speed bus 914 and the storage device 906. Each of the components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and can be attached to a common motherboard or mounted in other manners as required. The processor 902 processes instructions for execution within the computing device 900, including instructions stored in the memory 904 or the storage device 906, and can display graphical information for a GUI for external input / output devices such as a display 916 connected to the high-speed interface 908. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as required. Also, multiple computing devices 900 may be connected to provide multiple portions of a plurality of operations required for each device (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0077] The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is a volatile memory unit(s). In another embodiment, the memory 904 may also be a non-volatile memory unit(s). The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0078] Storage device 906 can provide large-capacity storage for computing device 900. In one embodiment, storage device 906 may be a computer-readable medium such as a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configured devices, or may include such a medium. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer-readable medium or a machine-readable medium such as memory 904, storage device 906, or memory on processor 902.

[0079] High-speed controller 908 manages the bandwidth-intensive operations of computing device 900, and low-speed controller 912 manages the low-bandwidth-intensive operations. Such an assignment of functions is merely an example. In one embodiment, high-speed controller 908 is coupled to high-speed expansion port 910 that can receive memory 904, display 916 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In this embodiment, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth (registered trademark), Ethernet, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or may be coupled to a network device such as a switch or router, e.g., via a network adapter.

[0080] As shown in the figure, computing device 900 can be implemented in many different forms. For example, it may be implemented as a standard server 920, or may be implemented multiple times in a group of such servers. Also, it may be implemented as part of a rack server system 924. Additionally, it may be implemented in a personal computer such as a laptop computer 922. Alternatively, components of computing device 900 may be combined with other components within a mobile device (not shown) such as device 950. Each of such devices may include one or more of computing devices 900, 950, and the entire system may be composed of a plurality of computing devices 900, 950 that communicate with each other.

[0081] Computing device 950 includes, among other components, a processor 952, a memory 964, input / output devices such as a display 954, a communication interface 966, and a transceiver 968. Also, a storage device such as a microdrive or other device may be provided in device 950 to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and some of the components may be attached to a common motherboard, or may be attached in other manners as required.

[0082] Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor may be implemented as a chipset of chips including a plurality of separate analog and digital processors. The processor may provide coordination of other components of device 950, such as, for example, a user interface, applications executed by device 950, and control of wireless communication by device 950.

[0083] Processor 952 can communicate with a user via a control interface 958 and a display interface 956 coupled to a display 954. The display 954 can be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 956 can include appropriate circuitry for driving the display 954 to present graphic information and other information to the user. The control interface 958 can receive commands from the user and convert the commands for transmission to the processor 952. Additionally, an external interface 960 can be provided to communicate with the processor 952 to enable short-range communication with other devices of the device 950. The external interface 960 can provide, for example, wired communication in some embodiments, or wireless communication in other embodiments, and multiple interfaces can also be used.

[0084] Memory 964 stores information within the computing device 950. The memory 964 can be implemented as one or more of a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). Also, an extended memory 974 can be provided and connected to the device 950 via an extended interface 972 that can include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 974 can provide additional storage space for the device 950, or can store applications or other information for the device 950. Specifically, the extended memory 974 can include instructions for performing or supplementing the aforementioned processes, and can also include secure information. Thus, for example, the extended memory 974 can be provided as a security module of the device 950 and programmed with instructions that enable secure use of the device 950. Additionally, secure applications can be provided via the SIMM card along with additional information, such as placing identification information in a non-hackable manner on the SIMM card.

[0085] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable or machine-readable medium such as memory, extended memory 974, or memory on processor 952 that may be received via, for example, transceiver 968 or external interface 960.

[0086] Device 950 may communicate wirelessly via a communication interface 966 that may include a digital signal processing circuit if necessary. The communication interface 966 may provide communication in various modes or protocols including, among others, GSM (registered trademark) voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000 or GPRS. Such communication may occur, for example, via radio frequency transceiver 968. Additionally, short-range communication may be performed such as using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 970 may provide additional navigation and location-related wireless data to device 950, which may be used as needed by an application running on device 950.

[0087] Device 950 may also perform voice communication using an audio codec 960 that may receive voice information from a user and convert it into usable digital information. The audio codec 960 may similarly generate sounds audible to the user through, for example, a speaker (such as within the handset of device 950). Such sounds may include sounds from a voice telephone call, may include recorded sounds (such as voice messages, music files, etc.), and may also include sounds generated by an application operating on device 950.

[0088] As shown in the figure, the computing device 950 can be implemented in many different forms. For example, the computing device 950 can be implemented as a mobile phone 980. The computing device 950 can also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile devices.

[0089] FIG. 10 shows an example of a general computer system 1000 that may be the gesture detection circuit 360 of FIG. 7 that can be used with the techniques described herein. The computing system 1000 is intended to represent various exemplary forms of large-scale data processing devices such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. The computing system 1000 may be a distributed system having multiple processors, which may include network-connected storage nodes interconnected by one or more communication networks. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0090] The computing system 1000 may include any number of computing devices 1080a - d. The computing devices 1080a - d may include servers or rack servers, mainframes, etc. that communicate via a local area network or wide area network, dedicated optical links, modems, bridges, routers, switches, wired networks or wireless networks, etc.

[0091] In some embodiments, each computing device may include a plurality of racks. For example, computing device 1080a includes a plurality of racks 1058a - 1058n. Each rack may include one or more processors such as processors 1052a - 1052n and 1062a - 1062n. The processors may include data processors, network-connected storage devices, and other computer control devices. In some embodiments, one processor may operate as a master processor and control scheduling tasks and data distribution tasks. The processors may be interconnected via one or more rack switches 1062a - 1062n, and one or more racks may be connected via switch 1078. Switch 1078 may handle communication between a plurality of connected computing systems 1000.

[0092] Each rack may include memories such as memories 1054 and 1064, and storage such as 1056 and 1066. Storages 1056 and 1066 may provide mass storage and may include volatile storage or non-volatile storage, such as a network-connected disk, floppy disk, hard disk, optical disk, tape, flash memory, or other similar solid-state memory device, or an array of devices including a storage area network or other configured devices. Storage 1056 or 1066 may be shared among multiple processors, multiple racks, or multiple computing devices and may include a computer-readable medium storing instructions executable by one or more of the processors. Memories 1054 and 1064 may include, for example, one or more volatile memory units, one or more non-volatile memory units, and / or other forms of computer-readable media such as magnetic or optical disks, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. A memory such as memory 1054 may also be shared among processors 1052a-1052n. Data structures such as indexes may be stored, for example, across storage 1056 and memory 1054. Computing system 1000 may include other components not shown, such as a controller, bus, input / output devices, communication module, etc.

[0093] The entire system may be composed of multiple computing devices 1000 that communicate with each other. For example, device 1080a may communicate with devices 1080b, 1080c, and 1080d, which may collectively be known as computing device 1000. As another example, gesture detection circuit 360 of FIG. 7 may include one or more computing devices 1000. Some of the computing devices may be located close to each other geographically, while others may be located far apart geographically. The layout of system 1000 is only an example, and the system may take other layouts or configurations.

[0094] The various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can be either special or general purpose, and can include one or more computer programs executable and / or interpretable in a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0095] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0096] To provide interaction with a user, the systems and techniques described herein are implemented on a computer having a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (such as a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback), and the input from the user can be received in any form including acoustic, speech language, or tactile input.

[0097] The systems and techniques described herein can be implemented on a computing system that includes backend components (such as a data server), or middleware components (such as an application server), or frontend components (such as a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing device that includes any combination of such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication of any form or medium (such as a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0098] A computing system can include a client and a server. The client and the server are generally far apart from each other and usually interact through a communication network. The relationship between the client and the server is created by computer programs that operate on respective computers and have a client-server relationship with each other.

[0099] Some embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0100] Also, when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, that element can be directly on, directly connected to, or directly coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, no intervening elements are present. Throughout the description of the embodiments, even if the terms directly on, directly connected, or directly coupled are not used, an element shown to be on, connected, or coupled can be so referred to. The claims of this application can be amended to enumerate the exemplary relationships described in this specification or shown in the drawings.

[0101] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Accordingly, it is to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. They are presented by way of example only and not by way of limitation, and it is to be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except mutually exclusive combinations. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

[0102] Additionally, the logical flow shown in the figures does not require the particular order shown, i.e., a sequential order, to achieve the desired result. Additionally, steps may be provided to the described flow, steps may be eliminated from the described flow, components may be added to the described system, and components may be eliminated from the described system. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. 1. A method comprising: capturing a sequence of images through the skin of a user's wrist; determining a biological fluid flow metric based on the sequence of images; and determining a gesture made by the user based on the biological fluid flow metrics; and triggering execution of a command related to an object displayed within an augmented reality (AR) system based on the gesture; A method comprising:

2. Determining the gesture formed by the user comprises: applying a first model that maps the sequence of images to the biological fluid flow metrics; applying a second model that maps the biological fluid flow metrics to the gestures; and The method of claim 1 , comprising:

3. the first model includes a convolutional regression neural network; The method comprises: training the convolutional regression neural network based on a dataset comprising a sequence of images and corresponding values of the biological fluid flow metrics; The method of claim 2 , further comprising:

4. The method of any one of claims 1 to 3, wherein the biological metric comprises a change in perfusion index between frames of the sequence of images.

5. Capturing the sequence of images in two dimensions through the skin of the user's wrist includes: emitting electromagnetic radiation from a radiation source to an interior of the user's wrist; receiving the electromagnetic radiation reflected from the interior of the user's wrist into a radiation detector; forming the sequence of images by sampling the electromagnetic radiation reflected from the interior of the user's wrist into the radiation detector at a specified frame rate; The method according to any one of claims 1 to 4, comprising:

6. The method of claim 5 , wherein the radiation source comprises an infrared (IR) projector mounted on a camera, the camera comprising the radiation detector.

7. the radiation source includes a left radiation source and a right radiation source; emitting the electromagnetic radiation from the radiation source to the interior of the user's wrist, emitting electromagnetic radiation from the left radiation source to an interior first portion of the user's wrist and from the right radiation source to an interior second portion of the user's wrist; The method according to claim 5 or 6, comprising

8. The method according to claim 7, wherein each of the left radiation source and the right radiation source includes a respective pair of infrared (IR) light emitting diodes (LEDs).

9. The radiation detector includes a left channel detector and a right channel detector, Receiving the electromagnetic radiation reflected from the interior of the user's wrist into the radiation detector Receiving the electromagnetic radiation reflected from the interior of the user's wrist by the left channel detector and the right channel detector The method according to any one of claims 5 to 8, comprising

10. Sampling the electromagnetic radiation reflected from the interior of the user's wrist Capturing respective images substantially simultaneously with each of the left channel detector and the right channel detector The method according to claim 9, comprising

11. An augmented reality (AR) system, comprising An image capture device configured to capture a sequence of images through the skin of a user's wrist, and A gesture detection circuit coupled to a memory, the gesture detection circuit being configured to Determine a biofluid flow metric based on the sequence of images, Determine a gesture formed by the user based on the biofluid flow metric, and Trigger the execution of a command related to an object displayed within the augmented reality (AR) system based on the gesture. An AR system configured to perform

12. The AR system according to claim 11, wherein the image capture device is disposed on a wristband worn around the user's wrist.

13. The image capture device Includes an electromagnetic radiation source configured to emit the electromagnetic radiation in the infrared (IR) wavelength band, and the image capture device further includes A detector configured to detect the electromagnetic radiation reflected from the interior of the user's wrist, the detector being configured to detect electromagnetic radiation within the IR wavelength band. The system according to claim 11 or 12.

14. An IR controller configured to control the emission of the electromagnetic radiation by the electromagnetic radiation source according to a schedule The AR system according to claim 13, further comprising

15. The AR system according to claim 13 or 14, wherein the electromagnetic radiation source and the detector are arranged at the same location within a single housing.

16. The detector includes a left channel detector and a right channel detector, The electromagnetic radiation source includes a first pair of IR light emitting diodes (LEDs) and a second pair of IR LEDs, the first pair of IR LEDs being configured to emit the electromagnetic radiation such that the electromagnetic radiation is received by the left channel detector, and the second pair of IR LEDs being configured to emit the electromagnetic radiation such that the electromagnetic radiation is received by the right channel detector. The AR system according to any one of claims 13 to 15.

17. A computer program product comprising a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit, causes the processing circuit to execute a method, the method comprising: capturing a sequence of images through the skin of a user's wrist; determining a biofluid flow metric based on the sequence of images; determining a gesture formed by the user based on the biofluid flow metric; triggering the execution of a command related to an object displayed within an augmented reality (AR) system based on the gesture. A computer program product.

18. Determining the gesture formed by the user includes: applying a first model that maps the two-dimensional sequence of images to the biofluid flow metric; applying a second model that maps the biofluid flow metric to the gesture. The computer program product according to claim 17.

19. The first model includes a convolutional regression neural network, The method further includes: training the convolutional regression neural network based on a dataset including a two-dimensional sequence of images and corresponding values of the biofluid flow metric. The computer program product according to claim 18.

20. The computer program product according to any one of claims 17 to 19, wherein the biological metric includes a change in a perfusion index between frames of the two-dimensional sequence of images.

Images