Head and eye based gesture recognition

Abstract

Aspects of the technology relate to using a wearable computing device 100 to determine and utilize head and eye poses. The wearable computing device can include an electrooculogram electrode 250 configured to provide first feedback to one or more processors 220, and an inertial measurement unit 260 configured to provide second feedback to the one or more processors. The first feedback can correspond to eye movement, and the second feedback can correspond to head movement. The wearable computing device can also include one or more processors that can be configured to use the first feedback and the second feedback to determine whether the head movement and the eye movement correspond to a trigger for an action, and cause the wearable computing device to perform the action based on the determination.

Description

[0001] Cross-referencing related applications

[0002] This application claims the benefit of U.S. Application No. 16 / 595,634, filed October 8, 2019, which is incorporated herein by reference in its entirety. Background Technology

[0003] In many aspects of modern life, computing devices such as personal computers, laptops, tablets, mobile phones, and countless network-connected devices are becoming increasingly prevalent. As computing devices become more ubiquitous, controlling them, both physically and environmentally, demands more natural ways of interaction. Common human gestures are often considered natural interactions, but they are prone to false positives. For example, using a "smiling gesture" to trigger turning on a light can lead to an unwanted trigger because the person is naturally smiling in the context. On the other hand, more complex gestures, such as moving fingers in different motion patterns, may have better recognition performance, but because they are not natural movements, they are difficult for people to remember or even execute correctly. In fact, many gestures that work well in laboratory settings are not suitable for general use. In other words, gestures need to be distinctive enough for computer devices to recognize them and for people to remember and execute them. Summary of the Invention

[0004] One aspect of this technology provides a system for determining and utilizing head and eye postures. The system includes a wearable computing device having electrooculography electrodes configured to provide first feedback to one or more processors. The first feedback corresponds to eye movements. The system also includes an inertial measurement unit configured to provide second feedback to the one or more processors. The second feedback corresponds to head movements. The system further includes one or more processors configured to use the first and second feedback to determine whether the head and eye movements correspond to triggers of an action, and to cause the wearable computing device to perform the action based on the determination.

[0005] In one example, the wearable computing device includes a nose pad, and the electrooculography electrodes are disposed at the nose pad. In another example, the wearable computing device includes one or more side support arms, and the inertial measurement unit is disposed at one of the one or more side support arms. In another example, the inertial measurement unit is a gyroscope. In yet another example, the trigger corresponds to the vestibulo-ocular reflex. In yet another example, the system further includes a memory storing a first classifier configured to determine whether the head movement and eye movement correspond to the trigger. In this example, determining whether the head movement and eye movement correspond to the trigger includes using the first feedback to generate a value, and the one or more processors are further configured to input the value into the first classifier to determine whether the head movement and eye movement correspond to the trigger. Additionally or alternatively, the memory also stores a second classifier configured to determine the posture type associated with the action. In this example, the one or more processors are further configured to input the value into the second classifier based on the determination to determine the posture type.

[0006] Another aspect of this technology provides a method for determining and utilizing head and eye postures. The method includes receiving, by one or more processors of a wearable computing device, first feedback corresponding to eye movements from electrooculography (EOG) electrodes, the EOG electrodes being part of the wearable computing device; receiving, by the one or more processors, second feedback corresponding to head movements from an inertial measurement unit, the inertial measurement unit being part of the wearable computing device; using the first and second feedbacks to determine whether the head and eye movements correspond to a trigger for an action; and having the one or more processors cause the wearable computing device to execute the action based on the determination.

[0007] In one example, the trigger corresponds to the vestibulo-ocular reflex. In another example, determining whether the head movement and eye movement correspond to the trigger includes using the first feedback to generate a value and inputting the value into a classifier that outputs an indication of whether the head movement and eye movement correspond to the trigger. In this example, the value includes a Pearson correlation between the first feedback and the second feedback. Additionally or alternatively, generating the value includes using a sliding window method on the first feedback. Additionally or alternatively, the method further includes, based on the determination, inputting the value into a second classifier to determine a pose type, wherein the pose type is associated with the action. In another example, determining whether the head movement and eye movement correspond to the trigger includes using the second feedback to generate a second value and inputting the second value into a classifier that outputs an indication of whether the head movement and eye movement correspond to the trigger. In this example, the value includes a Pearson correlation between the first feedback and the second feedback. Additionally or alternatively, generating the value includes using a sliding window method on the second feedback. Additionally or alternatively, the method further includes, based on the determination, inputting the value into a second classifier to determine a pose type, wherein the pose type is associated with the action. In another example, the inertial measurement unit is a gyroscope. Attached Figure Description

[0008] Figure 1 This is an example of a wearable computing device based on this technical aspect.

[0009] Figure 2 This is an example block diagram of a wearable computing device according to this technical aspect.

[0010] Figure 3 This is an example of comparing data based on this technical aspect.

[0011] Figure 4 This is an example of comparing data based on this technical aspect.

[0012] Figure 5 This is an example block diagram illustrating the determination and utilization of head and eye postures according to this technical aspect.

[0013] Figure 6 This is an example flowchart for determining and utilizing head and eye postures according to this technical aspect. Detailed Implementation

[0014] Overview

[0015] This technology generally involves determining and utilizing head and eye posture. The human eye has four modes of movement: saccades, smooth tracking, eye turning, and the vestibular-ocular reflex (VOR). This VOR serves as a representation of visual attention to initiate and determine posture and related commands. In this reflexive state, the human eye naturally compensates for head movement by rotating in a complementary manner. For example, when a person focuses their attention on a point while moving their head, there is a strong negative correlation between head rotation and eye position. This correlation can therefore be used as a platform for many head posture-based technologies.

[0016] One implementation may include a wearable computing device, such as a device that can be worn on the face. The wearable computing device may include a pair of glasses with a frame, one or more of the aforementioned displays, a central frame support, lenses, and side support arms. The wearable computing device may also include a battery, memory, processing hardware, and multiple sensors. One of the sensors may include an electrooculography (EOG) electrode that measures eye movements and provides feedback to the processing hardware. Another sensor may include an inertial measurement unit (IMU), such as a gyroscope, that tracks head rotation and provides feedback to the processing hardware.

[0017] EOG and IMU feedback can be used as triggers to determine possible postures. A first classifier can be used to determine whether the feedback from the EOG and IMU corresponds to a posture trigger. As mentioned above, if a person focuses their eyes on a point while moving their head in different directions, the feedback will be inverted. This could correspond to a posture trigger. If the first classifier indicates that the feedback represents a trigger, a second classifier can be used to determine the type of posture being performed. Similarly, values ​​including Pearson correlation can then be input into the second classifier to determine a specific posture. Postures can be associated with corresponding commands or actions, and once the action is determined, the wearable computing device can execute that action.

[0018] The features described in this paper can provide a way to recognize gestures and commands with minimal computational resources. Furthermore, compared to camera-based gesture recognition methods, such as those requiring an inward-facing camera to read an electrooculogram (EOG), which may have high battery costs, the features described in this paper can significantly reduce the probability of false positives, reduce power consumption, and recognize gestures faster. Other attempts to study gaze-based gestures focus on the idea of ​​synchronizing eye movements with moving stimuli. In this case, the eyes move in a smooth-tracking manner. However, for a typical person, the mental burden required to utilize VOR is less than that of smooth tracking due to the reflexive nature of VOR.

[0019] Example device

[0020] One implementation could include a wearable computing device, such as a device that can be worn on the face. For example, Figure 1 An example wearable computing device 100 is depicted, described herein as a pair of "smart" glasses. The wearable computing device may include a pair of glasses having frames 110, 112, a central frame support 120, lenses 130, 132, and side support arms 140, 142 that can be placed above the wearer's ears. Although not depicted in the example, the wearable computing device may also include a near-eye display, a head-mounted display (HMD), or a head-up display (HUD). Alternatively, the wearable computing device may include other types of head-mounted devices instead of glasses, such as earplugs, or a device including a front frame support, one or more of the aforementioned displays, and side support arms without lenses.

[0021] Figure 2 A block diagram of a wearable computing device 100 is depicted. In this example, the wearable computing device 100 includes processing hardware 210, which includes one or more processors 220, a memory 230 storing instructions 230 and data 232, a battery 240, multiple sensors (including an EOG 250 and an IMU 260), and a camera 270. One of the sensors may include an electrooculography (EOG) electrode or EOG 250, which measures eye movements and provides feedback to the processing hardware 210. As an example, these electrodes may be attached to or mounted within a nose pad 122 extending from a central frame support 120 of the wearable computing device. Another of the sensors may include an inertial measurement unit (IMU) or IMU 260, such as a gyroscope, which tracks head rotation and provides feedback to the processing hardware 210. As an example, as shown, the IMU may be attached to or mounted within a side support arm 140, or alternatively, within a side support arm 142. In the example of the earbud, the EOG and IMU can each be located on multiple parts of the earbud. Because the electrical signals detected by the EOG can propagate throughout the head, eye movements can still be detected at or around the ear. For example, the EOG can measure electrical signals from within the ear canal.

[0022] Memory 230 stores information accessible to processor 220, including instructions 232 and data 234 that can be executed or otherwise used by processor 220. Memory 230 can be any type capable of storing one or more processor-accessible information, including computing device-readable media, or other media that store data readable by electronic devices, such as hard disk drives, memory cards, ROM, RAM, DVDs or other optical discs, and other writable and read-only memories. Systems and methods can include different combinations of the foregoing, wherein different portions of instructions and data are stored on different types of media.

[0023] Instructions 232 can be any set of instructions that are directly executed by the processor (e.g., machine code) or indirectly executed (e.g., scripts). For example, instructions can be stored as computing device code on a computing device-readable medium. In this regard, the terms "instruction" and "program" are used interchangeably herein. Instructions can be stored in object code format for direct processor processing, or in any other computing device language, including scripts or sets of stand-alone source code modules that are interpreted on demand or compiled in advance. The functionality, methods, and routines of these instructions will be explained in more detail below.

[0024] Data 234 can be retrieved, stored, or modified by one or more processors 930 according to instructions 234. For example, although the claimed subject matter is not limited to any particular data structure, the data can be stored in a computing device register, in a relational database as a table with multiple different fields and records, an XML document, or a flat file. The data can also be formatted in any computing device-readable format. For example, the data can store information about the expected position of the sun relative to the earth at any given time, as well as information about the location of a network target.

[0025] Processor 220 can be any conventional processor, such as a commercial CPU or GPU. Alternatively, one or more processors can be dedicated devices, such as ASICs or other hardware-based processors. Although Figure 2 Processor 220, memory 230, and other elements of processing hardware 210 are shown functionally within the same box, but it should be understood that a processor or memory may actually include multiple processors or memories, which may or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage medium located in a housing different from the housing of processing hardware 210.

[0026] The processing hardware may also include one or more wired connections 240 and wireless connections 242 (e.g., transmitters / receivers) to facilitate communication with sensors and / or remote computing devices.

[0027] Example Method

[0028] Figure 3An example comparison of EOG and IMU (here, gyroscope) feedback is provided for saccade eye movements as a person moves their head to look around. For example, a person might look to one side while simultaneously turning their head in the same direction and back. In this example, eye movements are measured using voltage potentials, and head movements are measured using angular velocities. It can be seen that the head movements detected by the gyroscope have a slight delay compared to the eye movements detected by the EOG, because the eyes move slightly ahead of the head. In other words, the eye movements and head movements are roughly in the same direction. Therefore, the signals from the feedback (or more precisely, the voltage potentials and angular velocities) are correlated and in phase.

[0029] Figure 4 An example comparison of EOG and IMU (here, gyroscope) feedback under VOR eye movements is provided when a person moves their head to look around. For example, a person can focus on a point while simultaneously turning their head in a specific direction (left or right) and back. In other words, the eye movements and head movements are roughly in opposite directions. Therefore, the signals from the feedback (or more precisely, voltage potential and angular velocity) are uncorrelated and out of phase.

[0030] This inverse relationship between EOG and IMU feedback can be used as a trigger to determine possible poses. Figure 5 An example block diagram illustrating the determination and utilization of head and eye pose is presented. For example, feedback from EOG 250 and IMU 260 can be sent to processing hardware 210. Processor 220 can then process the feedback using a sliding window approach to determine multiple values. As an example, the window size can include 100 data points, and 5 data points can be "slid" across. Feedback within a window can be processed to determine multiple values, such as the maximum value of the window, the minimum value of the window, the range of the window, and the average value of the window. Furthermore, feedback for each window can be compared to determine the Pearson correlation for each window. Other values, such as cross-correlation (as an indicator of phase shift) and the delta-peak time of the EOG and IMU feedback, can also be determined. The delta-peak time can be determined by finding the maximum and / or minimum value of the feedback within the window (e.g., the peak value of the feedback). If the peak value is not at the beginning or end of the window, the delta-peak time can be determined from the difference between the timestamps of the peak value in the EOG feedback and the peak value in the IMU feedback.

[0031] Then, multiple values, including Pearson correlation and others, can be input into the first classifier 410 to determine whether the feedback from EOG 250 and IMU 260 corresponds to a gesture trigger. As described above, if a person moves his or her head in different directions while focusing their eyes on a point, the feedback between EOG 250 and IMU 260 will be out of phase. This out-of-phase determination can correspond to a gesture trigger. In this respect, classifier 410 can determine whether the feedback (or more precisely, the multiple values) indicates that the movement corresponds to or does not correspond to a gesture trigger. As an example, classifier 420 can output a representation corresponding to one or the other of a gesture trigger or not, such as a binary value, such as 0 or 1. In other words, the output can indicate whether the movement of the eyes and head corresponds to a gesture trigger. If classifier 410 determines that these values ​​do not correspond to a gesture trigger, processor 220 can continue processing additional values ​​from another window. In addition, the classifier 410 can be a decision tree, a random forest classifier, or other models that can be used to determine whether a feedback represents a trigger based on the amplitude of eye and head movements and the degree of irrelevance of the feedback.

[0032] If the first classifier indicates that the feedback represents a trigger, then the second classifier 420 can be used to determine the type of pose being performed. Similarly, multiple values, including Pearson correlations, can then be input into classifier 420 to determine a specific pose. Classifier 420 can be a decision tree, a random forest classifier, or other models that can be used to determine a specific pose. As an example, classifier 420 can output a representation corresponding to an identifier for a specific pose. In some instances, classifier 410 can be used to “segment” the data or find data windows where poses exist. Subsequently, classifier 420 can be a more complex classifier that can be used to identify multiple poses. For example, classifier 420 can utilize Dynamic Time Warp (DTW) or Structured Empirical Cumulative Distribution Function (sECDF) features to identify multiple poses with high accuracy.

[0033] Each of these gesture identifiers can be associated with a corresponding command or action stored in a lookup table, database, or other storage structure in memory 230. Examples of actions may include controlling the device, capturing an image or controlling aspects or features of the wearable computing device, requesting an online (e.g., via the Internet) search for visually similar images or objects or object types appearing in the field of vision, generating a tag or bookmark on an object of interest in the field of vision, establishing a social interaction on an object in the field of vision, etc. Once the action is determined, the wearable computing device can perform the action. Example gestures may include turning the head left and then right (e.g., back to the center), turning the head right and then left (e.g., back to the center), moving the head up and then down, moving the head down and then up, moving the head clockwise, moving the head counterclockwise, etc.

[0034] As an example implementation of the aforementioned features, a person may see a poster of interest and want to take notes on it. Wearing the wearable computing device 100, instead of taking out his or her smartphone, finding a note-taking app, and entering relevant details, he or she can visually focus on the poster, then tilt his or her head to the left and back to the center. The feedback from the EOG 250 and IMU 260 will be irrelevant, or completely opposite. Therefore, the wearable computing device 100 can use classifier 410 to determine that the person's VOR is engaged and that the eye and head movements correspond to the gesture triggers. The wearable computing device 100 can then use a second classifier to determine that a "tilt left" gesture was performed during the reflection. The wearable computing device 100 can then determine, based on the gesture, actions such as "on" or "off" and can automatically send commands to remote devices to turn lights (e.g., light bulbs) or other IoT (Internet of Things) devices on or near the point of interest. This simplifies the process of controlling devices and enables people to perform other complex tasks (e.g., turning lights on or off) more efficiently.

[0035] In some cases, sensors, such as the EOG 250 and IMU 260, do not need to be actually turned on and record sensor data or feedback until a certain condition is met. For example, the IMU 260 does not need to record any data until a minimum acceleration is detected, or in other words, the device has been shaken with a certain force. These detection features can be incorporated into the hardware of the IMU 260 itself to minimize power consumption. In response to the minimum acceleration being met, the IMU 260 can send a signal to the processor to indicate that the IMU is active (i.e., “on”). The processor 220 can then begin receiving and processing feedback from the IMU 260. In other examples, one or both of the classifiers 410 and 420 can be executed in the hardware of the IMU and / or EOG. In this regard, after a trigger is identified, the IMU 260 and / or EOG 250 can output a signal to the main processor determining that the trigger has been met and / or the gesture type. Thereafter, the processor can determine and execute the action associated with the gesture type.

[0036] Figure 6An example flowchart for determining and utilizing head and eye postures is provided, which can be executed by one or more processors of wearable computing device 100, such as one or more processors 220 of wearable computing device 100. In this example, at block 610, second feedback is received from electrooculography (EOG) electrodes corresponding to eye movements. The EOG electrodes are part of the wearable computing device. At block 620, second feedback is received from an inertial measurement unit (IMU) corresponding to head movements. The IMU is part of the wearable computing device and may include a gyroscope. At block 630, the first and second feedbacks are used to determine whether the head and eye movements correspond to triggers for an action. At block 640, the wearable computing device performs an action based on this determination.

[0037] The features described in this paper can provide a way to recognize gestures and commands with minimal computational resources. Furthermore, compared to camera-based gesture recognition methods, such as those requiring an inward-facing camera to interpret electrooculograms (EOGs) and potentially incurring high battery costs, the features described in this paper can significantly reduce false positives, lower power consumption, and recognize gestures faster. Other attempts to study gaze-based gestures focus on the idea of ​​synchronizing eye movements with moving stimuli. In this case, the eyes move in a smooth-tracking manner. However, for the typical human, the mental workload required to utilize VOR is less than that of smooth tracking due to the reflexive nature of VOR.

[0038] Most of the above-described alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be interpreted rather than limited to the subject matter defined by the claims. As an example, the preceding operations need not be performed in the exact order described above. Instead, multiple steps can be performed in different orders or simultaneously. Steps may also be omitted unless otherwise stated. Furthermore, the specifications of the examples described herein, and the phrases such as “for example,” “comprising,” etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended to illustrate one of many possible embodiments. Additionally, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A wearable device configured to detect head and eye movements of a user, the wearable device comprising: a body configured to be worn by the user; a first sensor coupled to the body and positioned to receive first feedback corresponding to a first movement of at least one eye of the user; a second sensor coupled to the body and positioned to receive second feedback corresponding to a second movement of a head of the user; a processor couplable to the first sensor and the second sensor; and a non-transitory computer-readable medium having instructions that, when executed by the processor, cause the processor to: determine, using a first classifier and based at least in part on the first feedback and the second feedback, whether the first movement and the second movement correspond to a representation of a gesture performed by the user; determine, using a second classifier and based at least in part on the first feedback and the second feedback, the gesture performed by the user; and perform an action associated with the gesture.

2. The wearable device of claim 1, wherein the first sensor comprises at least one electrooculogram electrode.

3. The wearable device of claim 2, wherein the first feedback comprises an electrical signal associated with the first movement.

4. The wearable device of claim 1, wherein the second sensor comprises an inertial measurement unit.

5. The wearable device of claim 4, wherein the inertial measurement unit comprises a gyroscope.

6. The wearable device of claim 1, wherein the first classifier is configured to determine that the first movement and the second movement correspond to the representation of the gesture when the first movement and the second movement are in different directions.

7. The wearable device of claim 1, wherein the body comprises at least one earpiece configured to be worn at least partially on or in an ear of the user.

8. The wearable device of claim 7, wherein the first sensor is positioned to receive the first feedback via an ear canal of the ear of the user.

9. The wearable device of claim 1, wherein the first classifier is configured to compare the first feedback to the second feedback to determine whether the first movement corresponds to a vestibulo-ocular reflex of the at least one eye of the user.

10. A method for controlling a wearable device using head and eye movements of a user, the method comprising: receiving, by one or more processors of the wearable device, first feedback from a first sensor of the wearable device, the first feedback corresponding to a first movement of at least one eye of the user; receiving, by the one or more processors, second feedback from a second sensor of the wearable device, the second feedback corresponding to a second movement of a head of the user; ​ ​ determining, using a first classifier, whether the first motion and the second motion correspond to a representation of a gesture performed by the user based at least in part on the first feedback and the second feedback; determining, using a second classifier, the gesture performed by the user based at least in part on the first feedback and the second feedback; and causing, by the one or more processors, the wearable device to perform an action associated with the gesture.

11. The method of claim 10, wherein the representation of the gesture comprises a trigger term of the gesture, and wherein using the first classifier comprises using the first classifier to determine whether the first motion and the second motion correspond to the trigger term of the gesture based at least in part on the first feedback and the second feedback.

12. The method of claim 11, wherein using the first classifier to determine whether the first motion and the second motion correspond to the trigger term comprises comparing the first feedback and the second feedback to determine whether the first motion corresponds to a vestibular ocular reflex of the user.

13. The method of claim 10, further comprising determining one or more values based at least in part on the first feedback and / or the second feedback, wherein using the first classifier comprises applying the first classifier to the one or more values.

14. The method of claim 13, wherein determining the one or more values comprises determining at least one of a Pearson correlation, a cross-correlation, and / or a delta-peak time based at least in part on the first feedback and / or the second feedback.

15. The method of claim 14, wherein using the second classifier comprises applying the second classifier to each of the one or more values when the first classifier determines that the first motion and the second motion correspond to the representation of the gesture.

16. The method of claim 10, wherein the first sensor comprises an electrooculogram electrode positioned to receive an electrical signal associated with a motion of at least one eye of the user, and wherein receiving the first feedback comprises receiving the electrical signal via the electrooculogram electrode.

17. A method for determining a head and eye gesture of a user using a wearable device, the method comprising: programming the wearable device to, via one or more processors of the wearable device: receive first feedback from a first sensor of the wearable device, the first feedback corresponding to a first motion of at least one eye of the user, and receive second feedback from a second sensor of the wearable device, the second feedback corresponding to a second motion of a head of the user; programming a first classifier of the wearable device to, via the one or more processors and based at least in part on the first and the second feedback, determine whether the first motion and the second motion correspond to a representation of a gesture performed by the user. ​ programming a second classifier of the wearable device to judge, via the one or more processors and based at least in part on the first and the second feedback, the gesture performed by the user; and programming the wearable device to perform an action associated with the gesture.

Images