Input methods for smart eyewear

Abstract

A computer-implemented method and device are disclosed for enabling natural motion input via a computing device, particularly those worn on the head. The disclosed technology involves defining application control events based on specific movements and positioning of a user's hands and fingers. Through the use of imaging data, the system can detect these movements and map key points on the user's hands and fingers to identify when predefined control events occur. Movements and positions detected in the imaging data are compared against those defined for application control events, enabling the system to generate corresponding application inputs. The method is capable of recognizing complex inputs, such as keystrokes and gestures, by interpreting hand and finger positioning relative to a virtual keyboard projected on the device's display. The user equipment device comprises a storage medium, a wearable display device, and one or more processors that execute instructions to implement the described method.

Description

INPUT METHODS FOR SMART EYEWEARInventors:Yue Zhang Hong Heather YuLiang PengXiyun SongZongfang LinYubin ZhouCLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 582,186, entitled “Finger Action Enabled Efficient and Real-time Text Entry for Augmented and Virtual Reality”, filed September 22, 2023, which is incorporated by reference herein in its entirety.FIELD

[0002] The disclosure generally relates to input techniques for virtual and augmented reality goggles and glasses.BACKGROUND

[0003] Human-computer interaction (HCI) typically occurs through any number of different input / output devices. Traditional input devices such as keyboards, mice, controllers and touchscreens, while effective in conventional computing environments, can disrupt the experience of augmented reality (AR) and virtual reality (VR) devices by requiring users to shift their attention away from the augmented environment. Typically, AR and VR devices include tracking technologies, such as outward facing two-dimensional and three-dimensional cameras, as well as other sensors, to accurately detect and interpret the physical environment of a wearer of such devices. These tracking technologies can be used to detect hand movements and gestures.

[0004] Human computer interaction in the form of text entry in AR / VR devices poses significant challenges due to the absence of physical keyboards. However, there are some limitations of these methods in real applications, such as physical fatigue, lack of haptic feedback, and low text entry efficiency.SUMMARY

[0005] One general aspect includes a computer implemented method of providing natural motion input to a computing device worn on a user’s head. The computer implemented method includes defining one or more application control events each of which generates an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user. The method also includes receiving imaging data over time from which movements and positioning of hands and fingers of the user can be determined and mapping multiple key points on the user’s hand and fingers. The method also includes calculating whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; and for each motion and position of the user’s hands and fingers matching a defined application control event, outputting, to an application on the computing device, a corresponding application input. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0006] Implementations may include the computer implemented method where the one or more application control events are defined by movement and positioning of hands and fingers relative to positions on a keyboard projected on a display of the computing device. Implementations may include any of the foregoing computer implemented methods wherein the control event is a keystroke entering a keyboard item. Implementations may include any of the foregoing computer implemented methods wherein the control event is a gesture selecting a word on a display of the computing device. Implementations may include any of the foregoing computer implemented methods wherein the keyboard items include one or more letters and further including calculating a word based on the one or more letters. Implementations may include any of the foregoing computer implemented methods wherein the method further may include repeating receiving and calculating to words selected by thegesture. Implementations may include any of the foregoing computer implemented methods wherein the calculating may include filtering at least two dimensions of the key points in the imaging data, and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions. Implementations may include any of the foregoing computer implemented methods wherein the filtering may include one or more of: calculating a moving average to key point position coordinates positions calculated over a given time window, calculating a dilatated difference between the key point position coordinates, and calculating a windowed standard deviation of key point coordinates. Implementations may include any of the foregoing computer implemented methods wherein key point coordinates are represented by two-dimensional coordinates. Implementations may include any of the foregoing computer implemented methods wherein key point coordinates are represented by three-dimensional coordinates.

[0007] One general aspect includes a user equipment device. The user equipment device includes a storage medium may include computer instructions and a wearable display device. The device also includes one or more processors coupled to communicate with the storage medium, where the one or more processors execute the instructions to cause the device to: define one or more application control events each of which generates an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user; receive imaging data over time from which movements and positioning of hands and fingers of the user can be determined; map multiple key points on the user’s hand and fingers; calculate whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; and for each motion and position of the user’s hands and fingers matching a defined application control event, output a corresponding application input to an application on the user equipment device.

[0008] Implementations may include a user equipment device where the one or more application control events are defined by movement and positioning of hands and fingers relative to positions on a keyboard projected on the display device.Implementations may include a user equipment device of any of the foregoing embodiments wherein the control event is a keystroke entering a keyboard item. Implementations may include a user equipment device of any of the foregoing embodiments wherein the control event is a gesture selecting a word on the display of the computing device. Implementations may include a user equipment device of any of the foregoing embodiments wherein the keyboard items include one or more letters and further including calculating a word based on the one or more letters. Implementations may include a user equipment device of any of the foregoing embodiments wherein the instructions cause the device to repeat receiving and calculating to words selected by the gesture. Implementations may include a user equipment device of any of the foregoing embodiments wherein the instructions cause the device to calculate by: filtering at least two dimensions of the key points in the imaging data; and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions. Implementations may include a user equipment device of any of the foregoing embodiments wherein the filtering comprises one or more of: calculating a moving average to key point position coordinates positions calculated over a given time window; calculating a dilatated difference between the key point position coordinates; and calculating a windowed standard deviation of key point coordinates. Implementations may include a user equipment device of any of the foregoing embodiments wherein the key point coordinates are represented by two-dimensional coordinates. Implementations may include a user equipment device of any of the foregoing embodiments wherein the key point coordinates are represented by three-dimensional coordinates.

[0009] One general aspect includes a non-transitory computer-readable medium storing computer instructions recognize human motion as input for an application. The non-transitory computer-readable medium storing computer instructions include defining one or more application control events each of which generates an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user. The instructions also include receiving imaging data over time from which movements and positioning of hands and fingers of the user can be determined. The instructions also include mapping multiple keypoints on the user’s hand and fingers. The instructions also includes calculating whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; and for each motion and position of the user’s hands and fingers matching a defined application control event, outputting, to an application on the computing device, a corresponding application input.

[0010] Implementations may include the non-transitory computer-readable medium where the one or more application control events are defined by movement and positioning of hands and fingers resembling input on a keyboard. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein control event is a keystroke entering a keyboard item. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein the keyboard items include one or more letters and further including calculating a word based on the one or more letters. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein the control event is a gesture selecting a word on a display of the computing device. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein the steps further may include repeating receiving and calculating to words selected by the gesture. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein calculating may include: filtering at least two dimensions of the key points in the imaging data, and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein filtering may include one or more of: calculating a moving average to key point position coordinates positions calculated over a given time window, calculating a dilatated difference between the key point position coordinates, and calculating a windowed standard deviation of key point coordinates. Implementations may include the non-transitory computer-readable medium of any of the foregoing embodiments wherein key point coordinates arerepresented by two-dimensional coordinates. Implementations may include the non- transitory computer-readable medium of any of the foregoing embodiments wherein key point coordinates are represented by three-dimensional coordinates.

[0011] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the Background.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying FIG.s for which like references indicate the same or similar elements.

[0013] FIG. 1A illustrates a typical environment in which the human input motion recognition technology may be utilized.

[0014] FIG. 1 B illustrates a user’s perspective view of their hands in an AR / VR device.

[0015] FIG. 1 C illustrates a user’s perspective view of their hands in an ARA / R device with a keyboard projected behind the user’s hands by the AR / VR device.

[0016] FIG. 2 is a flowchart illustrating a general method in accordance with the human input motion recognition technology.

[0017] FIG. 3 is a flowchart illustrating a mapping of letters typically found on a QWERTY keyboard to the fingers of a human hand.

[0018] FIG. 4 is a flowchart illustrating an interaction between a user and an application running on an augmented or virtual reality device in accordance with the human input motion recognition technology.

[0019] FIG. 5 is a flowchart illustrating a first method performed by an application to implement the human input motion recognition technology.

[0020] FIG. 6 illustrates one embodiment of a joint mapping using a type of hand land marking technology.

[0021] FIG. 7 is a flowchart illustrating a method for calculating the position of a hand and finger to determine an input on a virtual keyboard.

[0022] FIG. 8 illustrates an example of finger typing event detection data for five fingers based on X-axis and Y- axis positions.

[0023] FIG. 9 is a table illustrating the precision of detection of figure movement detection in accordance with the natural movement input technology.

[0024] FIG. 10 illustrates another typical environment in which the human input motion recognition technology may be utilized where a user’s hands are in free space.

[0025] FIG. 11 is a flowchart illustrating a second general method where control inputs include gestures in accordance with the human input motion recognition technology.

[0026] FIG. 12 is a flowchart illustrating second performed by a human input motion recognition application to implement the human input motion recognition technology.

[0027] FIG. 13 is a flowchart illustrating a third performed by a human input motion recognition application to implement the human input motion recognition technology.

[0028] FIG. 14A illustrates a calibrated stereo vision framework.

[0029] FIG. 14B and 14C illustrate X and Z axis detection of index finger motion using a stereo vision sensor system.

[0030] FIG 15 illustrates a set of inputs for each of the digits of a hand of, for example, the x-axis over time.

[0031] FIG. 16 illustrates an artificial neural network which may be utilized for hand and finger motion detection.

[0032] FIG. 17 illustrates an exemplary network device which may implement the technology described herein.

[0033] FIG. 18 illustrates one type of haptic feedback system suitable for use with the described technology.

[0034] FIG. 19 is a block diagram illustrating an energy harvester and controller suitable for use with the system of FIG. 18.WRITTEN DESCRIPTION

[0035] The present disclosure and embodiments present systems and methods for human input motion recognition technology suitable for use with an augmented reality (AR) and / or virtual reality (VR) headset (an “AR / VR device”). In embodiments, the technology is enabled by an AR / VR device worn by a user which interprets arm, hand and finger movements as inputs. In embodiments, the inputs comprise a typing input where a user rests their hands on a surface and “types” as one would normally type on a physical keyboard, though no physical keyboard need be used. In embodiments, the inputs comprise typing inputs in free space, where a user makes finger motions to “type” as one might normally type on a physical keyboard, but such motions occur in free space and not relative to a surface. In embodiments, the inputs comprise control gestures that are defined to provide controls to an application executing on the AR / VR device. This technology facilitates intuitive and efficient interaction with head-worn computing devices, enhancing user experience in various applications. The described technology utilizes at least one RGB camera to capture the motion of multiple fingers through continuous video frames. The technology presented herein allows for natural finger movement for text entry in AR / VR applications by assigning distinct alphabets to individual fingers and activates input through actions of finger movement. This capitalizes on the spatial characteristics of human fingers to distinguish different alphabets, reducing the complexity of alphabet recognition. The technology therefore provides advantages such as reduced physical fatigue, haptic feedback for an immersive experience, and improved text entry efficiency for inputs to a AR / VR application.

[0036] FIG. 1A illustrates a typical environment in which a user 100 is present in an environment while wearing an AR / VR device 150. Where the AR / VR device 150 comprises an augmented reality device to be worn over the eyes of a user, the device allows the user to view the real world objects through lenses of the device. In embodiments, device 150 may include a display integrated with the lenses through which the user views the environment. Where AR / VR device 150 is an augmented reality device, the device can overlay digital information, images, graphics, orholograms onto the user's view of the real world through the lenses. Where AR / VR device 150 is a virtual reality device, the user’s view is entirely focused on a display which blocks the user’s view of the user’s environment. In a VR device, the environment may be shown on the display as photorealistic video or may be replaced by alternative, rendered virtual environments. The AR / VR device 150 may include built-in speakers, microphones, sensors (such as gyroscopes, accelerometers, and magnetometers) and imaging sensors such as cameras (including RGB and depthsensing cameras). In embodiments, AR / VR device 150 may include an integrated network processing device or may be connected to a separate, external processing unit (as illustrated in FIG. 17). As used herein, the term “wearable device” refers to the AR / VR device 150 and its associated network processing device. AR / VR device 150 may provide various types of other input methods such as voice commands, hand gestures, and eye tracking, and has various types of connectivity to other network processing devices. AR / VR device 150 may include an internal or externally connected power source.

[0037] In the illustration of FIG. 1A, the user 100 may be looking at the user’s hands 130 resting on a surface 110. FIG. 1 B illustrates the user’s hands when viewed from the perspective of the user’s vision though an AR / VR device. In an AR device, the view may be an optical or a video pass through view of the user’s hands on the surface 110, while in a VR device, the view may be presented on a display screen in front of the user’s eyes and may be a display of the imaging sensor inputs of the device or a virtual rendering based on the sensor inputs of the device. FIG. 1 C illustrates the view of the user’s hands 130 in an AR / VR device with a keyboard projected within the viewing display of an AR / VR device. Such a keyboard projection may comprise the user’s actual hands shown through a pass through display with the keyboard rendered on the AR device display, or a rendering of the user’s hands in a VR device along with a rendering of the keyboard on the VR display. In embodiments, other control devices besides keyboards may be rendered and may include a joystick, buttons, or other control devices.

[0038] In FIG. 1A, the user’s hands 130 are resting on a surface 110. In embodiments, the technology tracks user hand movement relative to a surface onwhich the user’s hands are resting. In other embodiments, the technology tracks user hand and arm movement in free space (i.e. while not resting on a surface).

[0039] As illustrated in FIG. 1 C, a virtual input device such as a virtual keyboard 140 may be projected onto the display of the AR / VR device 150 so as to appear on the surface 110 where the user 100 is resting their hands 130. This allows the user to use hand and figure motion to input text by simulating “typing” on the virtual keyboard.

[0040] FIG. 2 is a flowchart illustrating a general method in accordance with the natural movement input technology. At 220, hand and figure motion data is acquired from one or more sensors 210. In embodiments, the motion sensing data may be provided by resistance or strain gauges on wearable implements on the user’s hands, capacitive or inductance devices on wearable implements, RGB cameras incorporated into the ARA / R device, stereo vision cameras incorporated into the ARA / R device, depth-camera systems incorporated into the ARA / R device, and / or laser systems incorporated into the AR / VR device (using, for example, laser speckle patterns, time of flight techniques or interferometric techniques to determine finger position and motion).

[0041] In accordance with the described technology, movement of the user’s hands and fingers is detected by the vision systems of the AR / VR device. In one embodiment, only RGB cameras from the AR / VR device are used. In another embodiment, RGB, stereo and / or depth-sensing cameras are used. In another embodiment, only depthsensing cameras are used. The technology thus presents embodiments which allow for determination of control inputs to applications on the AR / VR device using limited sensor systems of an AR / VR device.

[0042] At 230, the motion data is used to determine movement of the user’s fingers and hands and calculate a control input from the motion at 240. The input may be in the form of a gesture, with the gesture having a significance such as an object or word selection, moving a virtual object within the AR / VR device display, and / or enabling or disabling a feature of an application running in the AR / VR device. The gesture may be made relative to the surface 110 or in free space, without relation to a surface. For example, the gesture may include typing on the virtual keyboard 140 positioned on thesurface 110. In embodiments, the gesture may include “typing” in free space. The gesture may include only finger motion or include motion of the user’s hands and / or arms.

[0043] At 240, the input is translated into an output, which may comprise text, a control gesture or other action. In example embodiments described herein, text and gesture based inputs derived from visual input devices on the AR / VR device are described. At 250, feedback may be provided both relative to the motion detection at 230 and the control input of 240. In other embodiments, no feedback is provided. An exemplary type of feedback is a touch, vibration, light or sound associated with typing as is well known in mobile devices. In embodiments, the feedback may be provided by the AR / VR device or another wearable device associated with the AR / VR device. In embodiments, the feedback may be provided when a typing event by a finger is detected. A feedback actuator may be triggered to execute a set of visual, haptic and / or sound types of feedback. The type of feedback is configurable per user preferences. Haptic feedback may be provided by the user’s touching the surface during typing. Visual feedback may include highlighting the keyboard layout based on the finger movement for a short period. Sound feedback may comprise a brief sound based on finger movement. The duration and strength of the feedback signal may also be adjusted based on user preferences. At 260, the method loops to step 200 if additional input occurs, and if not, the method ends at 270.

[0044] Initial embodiments are described wherein sensors on the AR / VR device 150 are used to provide typing input relative to a two-dimensional surface on which the user, such as user 100 in FIG. 1A, is resting their hands. To detect typing events, a user’s hands 130 and fingers are mapped to likely characters the user would type with the hand and finger on a keyboard format. In embodiments, the mapping may be customized for each user so that individual user typing mechanics can be recognized. Typing events are detected and an output generated based on hand and finger movements associated with the mapping of the keys to the keyboard target locations. When typing, the user can put their hands on any surface that can provide a comfortable support, which reduces the physical fatigue. To type a word, a phrase, a sentence, or a paragraph, the user moves the finger which contains the letters of theword in order. After a sequence of movements of multiple fingers, an output text estimation module (1762) will estimate the user inputs and send it to the display of an AR / VR device. The user can select the displayed word, phrase, sentence, or paragraph to enter; decide to make any corrections or modifications on word, phrase, sentence or paragraph level during or after one typing event stops; or start the input of next word, phrase, sentence, or paragraph. The human input motion recognition technology may incorporate multi-finger actions for certain commands or as control signals. For instance, when all 10 fingers are motionless for a predefined period of time T, the system may acknowledge it as 'the end of typing session.’ An optional feedback actuator maybe used to highlight the keyboard layout of the typing finger on monitor, play a feedback sound through the speaker, or via other means.

[0045] FIG. 3 is a diagram depicting a mapping of letters and numbers to human fingers on hands 230. In the diagram, the letters are arranged in a mapping equivalent to a QWERTY key layout, but it should be recognized that the layout and mapping of letters and numbers may be in any number of different keyboard layouts depending on user preference and the user’s language. In embodiments herein, other keyboard formats may be selected by the user and maintained as a user preference associated with the user as part of a user profile and user preference database. As illustrated in FIG. 3, each finger is associated with a set of letters, numbers and / or control keys arranged in rows similar to the number, letter or control key’s position on the QWERTY keyboard layout (with the exception of the control CNTL and ALT keys). Rows 305 and 350 are associated with a user’s left and right pinkies, respectively. Rows 310 and 345 are associated with a user's left and right ring fingers, respectively. Rows 315 and 340 are associated with a user's left and right middle finger, respectively. Rows 320 and 335 are associated with a user's left and right index fingers, respectively, and rows 325 and 330 are associated with a user's left and right thumbs, respectively. As described herein, when a user types, the user’s hands 230 and fingers will reach the relative positions of the keys as laid out in the mapping shown in FIG. 3 (when a QWERY keyboard format is used). Positioning of the user’s hands can be aided by the projection of the virtual keyboard 140 in FIG. 1 C. As the user moves their hands and fingers, a determination is made as to the user’s intended key input and this istranslated into letters, numbers or control inputs by the system described herein. In order to determine the intended target key based on the key mapping shown in FIG. 3, a keyboard calibration, as discussed below, is used.

[0046] In alternative embodiments, the positions of the keys on the virtual keyboard need not be mapped to individual fingers. Many users type with less than all fingers of their hand and for such users, the technology can track the position of the fingers actually used by the user to generate input relative to the positions of the letters in the keyboard. Each mapping may be stored and associated with an individual user.

[0047] FIG. 4 illustrates an interaction between a user and human input motion recognition technology application running on AR / VR device. At 405, a user will rest their hands on a surface 110 while wearing an ARA / R device and may or may not view their hands using an ARA / R device. At 410, optionally, a virtual keyboard may be displayed to the user in the AR / VR device. In embodiments, the display is similar to that shown in FIG. 1 C. In other embodiments, no virtual keyboard need be displayed. At 415, a user will “type” by moving their hands and fingers to engage with keys associated with letters, numbers or control keys based on the keyboard mapping in use. At 420, the human input motion recognition technology will detect the hand and finger motion and, at 425, a text estimation module will compute the input to be recognized. Generally, text estimation comprises determining the letter, number, and / or control keys input, and optionally predicting a word generated by a combination of letters where the letters which are input do not exactly match a particular word (as in the case where a word is misspelled or an incorrect letter is detected). In addition to detecting finger implements, text estimation at 425 (as well as other text estimation steps herein) take advantage of existing algorithms widely used in smartphones or tablets, such as “text 9” (or “T9” -- a predictive text technology originally developed by Tegic Communications) or others. As shown at FIG. 3, several letters and digits are associated with each finger. When finger movement is detected, typing action is detected, and the sequence of finger movements are recorded, analyzed, and associated with letters. Text estimation takes advantage of knowledge databases (such as a dictionary) and machine learning algorithms, word, phrase, sentence, and / or paragraph level text sequences can be predicted and displayed on the monitoras suggestions for the user. Similar to other existing algorithms, text estimation can gain familiarity with the words and phrases the user commonly uses to improve both the accuracy and the speed of output text estimation. At 430, the predicted input in the form of text or a word prediction is output to the AR / VR display for review by the user. The text output may also be output at 445 to an application which may be using the text such as a word processing application, messaging application or the like on the ARA / R device. At 435, the user may select the text or continue typing. The selection of text at 435 may comprise using the keyboard keys in a combination to select the text, or making a text correction gesture (such as, for example, by “pointing” to the word in the ARA / R device display using the user’s finger to highlight the text). In embodiments, such selection comprises a control input that indicates that the user wishes to further manipulate the text. If the user does not select text (or engage other manual correction) at 435, then the method loops to 415 to continue motion detection. If the user selects all or some portion of the text output at 435, then the user may correct the text by moving their fingers in a typing motion at 440, and steps 420, 425, and 430 are repeated, resulting in a text input over the selected text or word. The corrected text is output to the AR / VR device display at 430 and to an application at 445 on the AR / VR device. The method continues accepting input until the user gestures to stop typing input at 450.

[0048] FIG. 5 illustrates a first method performed by human input motion recognition application to implement the natural movement input technology. At 502, motion data is acquired by imaging sensor devices of the AR / VR device. This can include acquiring visual data from single or multiple RGB or RGBD (RGB depth) cameras, stereo cameras, depth cameras or the like. At 504, hand pose estimation occurs. Hand pose estimation extracts hand 2D or 3D coordinates from the image based data inputs and determined the spatial configuration of a human hand captured by imaging sensors. Regions in the image or video frames that potentially contain a hand are identified using object detection techniques. Once the hand region is detected, segmentation can be applied to isolate the hand from the background and specific key points on the hand, such as joints or fingertips are identified. The spatial configuration of the hand is then determined based on the key points detected. In oneimplementation where the hands are on a surface, hand pose estimation at 504 generates the 2D coordinates of each joint. The X- and Y- -coordinates of each fingertip indicate their locations in each frame. This allows the system to easily track each fingertip and adjust the keyboard layout accordingly to match the positions. To detect finger movement, step 504 includes detecting hands in each frame, and outputting the 2D coordinates of hand joints. The 2D coordinates of each joint are concatenated across multiple continuous frames to obtain the coordinate series of each joint over a period of time. In other embodiments, 3D coordinates are used. One processing framework suitable for hand pose estimation is MediaPipe, an open-source framework from Google for building multimodal, applied machine learning pipelines.

[0049] At 512, a keyboard layout with fingers is (optionally) generated for display. The keyboard layout may be selected from a stored set of keyboard layouts 506 and may be selected 508 by a user or automatically selected based on a user profile having a stored keyboard layout preference. Step 508 may thus include receiving user input on the selected keyboard or automatically selecting the layout from the stored set 506. User profile and preferences store 510 may store user preferences including the keyboard layout, as well as hand sizes and keyboard layout calibration data. At 505, keyboard layout calibration is performed to ensure that the virtual keyboard layout aligns with the user hands and finger on the ARA / R display. This calibration may take the form of mapping the keyboard projection to a coordinate system which includes the user’s hands and prompting the user to “type” individual keys for specific letters, numbers or control characters. Using the finger and hand movements resulting from the prompts, the calibration 505 can adjust for the position of the user’s hands and fingers so that future positioning of user movements results in the intended letter, number or control key to be recognized. Calibrations can be stored along with associated keyboard layouts in a user profile and preferences store 510 for later use. In embodiments, user profile and preferences store includes a user pose to keyboard mapping 511 of the locations of the user’s hands relative to the key mapping positions of the keyboard in use. With reference to FIG. 3, such mapping would include the x / y coordinate positions of the user’s hands and fingers relative to the keys associated with each finger, so that movement of the user’s finger to an x / y position (in conjunctionwith a detected event indicating the user intends to type or press that key, discussed below) generates an input for that letter.

[0050] Once the virtual keyboard is generated at 512, event detection 514 detects movement of the user’s hand and fingers intended to create an input. This includes typing motion and gestures enabling selection of letters, words, phrases or text, or gestures to start or stop a text entry operation. When a typing event is detected at 514, text estimation at 425 occurs. The typing event at 514 detects which finger moved and outputs the characters or actions associated with the moved finger, based on the finger-alphabet layout map. Text estimation with respect to fingers on a surface is discussed further with respect to FIGs. 6 and 7. The event detection may (optionally) generate haptic, light and / or sound feedback 516. As discussed above, the feedback may be a vibration associated with typing, a sound may be associated with typing each letter, highlighting keys on the virtual keyboard in the display, or other types of feedback alerting the user that input is being generated. The type of feedback generated at 516 may be specified by the user as part of a user profile and preferences and may be stored in association with the user at store 510.

[0051] At 520, the estimated text is provided to the display screen of the AR / VR device 150 and to an application on the device which takes text as input. Incorrect estimation of text outputs may be manually or automatically corrected using machine learning algorithms. At 522, a determination is made as to whether a correction event has occurred either after text has been output at 520 or based on a correction gesture event detected at 514. If not, the method returns to step 514 for the next event. If so, the method processes a correction event at 524. The correction event may be the selection of a word or multiple words and may include determining additional manual input from the user from typing at 526 and / or include providing spelling, grammatical or other text suggestions to the display which may be selected by the user in the virtual interface and which replaces the selected text output provided at 520. If no correction event occurs at 522, at 526 the method continues determining whether additional typing is occurring. If no additional typing occurs, a determination is made as to whether an automated modification event should occur at 528. An automatic modification may occur when machine learning or other text correction algorithmsdetermine that a word is misspelled, or that the text estimated and word produced at 425 is likely incorrect in the context of other words the user has typed. Intelligent auto modification of the words or grammar of the text output may include a prompt by the system for with suggested corrections which may be selected by the user after the prompt, or an auto-correction of misspelled words or incorrect grammar. The method ends at 532 when no further input or corrections are made.

[0052] FIG. 6 illustrates one possible landmarking of key points for hand pose estimation. Two models may be used: one for palm detection and the other uses hand detection. The hand landmark model detects the key point localization of 21 handknuckle coordinates within the detected hand regions. Following a modeling of a user’s hands, a model specific to the user can be stored in the user profile and preferences database 510. Each of the landmarks illustrated in FIG. 6 can move in two or three physical dimensions, and a time domain. Embodiments of event detection may use two dimensions and the time domain, or three dimensions and the time domain, as described herein. In embodiments herein, an input may comprise a single image frame from a camera, with the output being a normalized 3D coordinate (X, Y, and Z) of 21 joints for each hand. As cameras provide video data, each video (including multiple frames), a 1 D time series data in each axis for each joint which can be processed (e.g. filtered or smoothed) to improve consistency and robustness.

[0053] FIG. 7 illustrates a method for determining the position of individual fingers and the letters typed by a user evaluating two dimensions of movement relative to a surface 100. In the embodiment of FIG. 7, two dimensions are used to track the end landmark of each finger (i.e. landmarks 4, 8, 12, 16 and 20). For each finger, the X and Y coordinates (relative to a defined x-y coordinate system defined by the ARA / R device) are tracked. In other embodiments, more than just the end landmark of each finger may be tracked. For each finger, at 705 and 725, the X and Y axis coordinates of end landmark are detected over time. At 707 and 727, one or more signal processing techniques are applied to the key point data. Such signal processing techniques may include, at 710 and 730, respectively, applying a moving average of the X and Y positions calculated over a given time window size. A moving averagesmooths the data to reduce noise and short-term fluctuations. In one embodiment, the moving average is calculated by:where C is the input series of the X or Y coordinate, 1 / V is the window size of the moving average filter, and MA is the de-noise series.

[0054] Such signal processing techniques may include, at 715 and 735, respectively applying a dilatated difference between the datapoints. The dilatated difference calculation highlights changes or trends over longer intervals by comparing values that are not immediately adjacent. To measure the coordinate change in a short time period, the event detection algorithm calculates the coordinates difference between two adjacent frames. For example, if a finger does not move in a last period, the coordinates difference between two nearby frames will close to 0. However, the range of finger movement is small in the camera view, and the finger movement speed is low in a 30 FPS camera video for example. Direct comparison of the coordinate difference between two adjacent frames may not clearly reflect the finger movement, especially with a noise coordinate (from hand pose extraction). Thus, the dilated difference between two frames may be calculated by:where T is the dilation distance in the time domain.

[0055] Such signal processing techniques may include, at 720 and 740, respectively calculating a windowed standard deviation of the datapoints. This measures the variability or dispersion of the dilated differences within a specified window. In one embodiment, the Standard deviation is calculated by:Where Wsis the STD windows size and DC[i]) ’represents the average value in the STD window.

[0056] Each of these signal processing techniques may be used alone or in combination with one or more of the other techniques to evaluate peak events in the X and Y coordinate data.

[0057] At 745, a weighted fusing of the X and Y coordinates occurs. The weighted fuse combines X and Y data to predict whether an action has occurred. For each letter, additional weight may be given to the X or Y position data to aid in differentiating one finger from other fingers. In one embodiment, the weighted fusion is given by:FU[i] = a • STDX[i] + (1 - a) • STDY [i], 0 < a <1 where a is the fusion weight of the X coordinate. The weighting assigns a different importance (weight) to one of the two coordinates before they are fused, to determine a position and letter position. At 750, peak detection over a threshold occurs. Peak detection filters out the noise associated with each set of data and in embodiments, selects peaks over a threshold which indicates that a user is attempting to type a letter. When a peak in the fused sequence surpasses a predefined threshold, it is identified as a typing event, indicating a significant changes of the hand landmark coordinate, and triggering a typing action. At 755, the specific letter associated with the peak is resolved from the X and Y coordinates associated with each peak in relation to an event frame index tying the X and Y coordinates together in time, with the X-Y position at the determined peak and the mapped position of the letter (in X-Y space in this embodiment) yielding a predication of each letter. The letter may be added to a word at 760. Using machine learning techniques as well as characteristics of the user profile, words can be suggested at 765 based on the identified letters, and the context to the word in relation to other words output by the method.

[0058] FIG. 8 illustrates graph of the X and Y positions of each finger, and the calculated peak event resulting from the method of FIG. 7, during a sample detection process. Columns 805 and 815 illustrated the X and Y coordinate positions observedby visual cameras for the respective thumb, index, middle, ring and pinky fingers. The vertical axis of columns 805 and 815 show the relative position in the X or Y coordinate while the horizontal axis of each column is the time in milliseconds. Columns 810 and 820 illustrate detected events (marked with an “x”) for each calculation. In the example of FIG. 8, only thumb events are detected.

[0059] FIG. 9 illustrates exemplary f-scores reflecting a measure of the method’s accuracy in predictive performance, considering precision and recall for an example test of the method discussed above. Precision measures how many of the predicted positive instances were actually positive, while recall measures how many of the actual positive instances were correctly predicted. A high precision score means that the model has a low rate of false positives, while a high recall score means that the model has a low rate of false negatives. Two camera positions representing cameras were utilized. The table presents the harmonic mean (F1 ) of the precision and recall calculated as:Precision x Recall Precision = 2 X - —Precision + RecallWhere.# of True PositivesPrecision(# of true positives) + (# of false positives)> .. # o f True PositivesRecall = - .(# of true positives) + (# of false negatives)

[0060] As shown, F1 scores of 0.9 (rounded) and higher demonstrate a high accuracy of the method.

[0061] In other embodiments, the human input motion recognition technology may be utilized with user hand motions not restrained by a surface 110. FIG. 10 illustrates a user 100 viewing the user’s hands in free space. In embodiments, the method detects input motions and gestures of the user’s hands in free space using sensorsassociated with an AR / VR device 150. For a user moving their hands in free space, the appearance of the user’s hands may resemble the depiction of FIG 1 B.

[0062] FIG. 11 is a flowchart illustrating a general method in accordance with the human input motion recognition technology for estimating a control input in free space, where the control input may be text input or another control gesture. At 1105, a user 100 moves their hands in free space while wearing an ARA / R device having visual or other imaging sensors. The motions of the user at 1105 may comprise typing movements or movements intended to create a control input. At 1110, the ARA / R device may display a keyboard or other virtual control device. In embodiments, user 100 in FIG. 10 may perceive a keyboard projected on the display of device 150 where the keyboard is in free space. In embodiments, a virtual control device may comprise a virtual joystick or other controller. In embodiments, the display is similar to that shown in FIG. 1 C except no surface would be displayed (since the user’s hands are not resting on a surface). In other embodiments, no virtual keyboard or virtual control device need be displayed. At 1115, a user will move their hands and fingers to generate a control input or “type” (by moving their hands and fingers to engage with keys associated with letters, numbers or control keys) based on the keyboard mapping or controller displayed. At 1120, the human input motion recognition application will detect the motion and at 1125, an estimation module will compute the input (control or text) recognized. At 1130, the predicted input in the form of a control input, text or a word prediction is output to the display of the AR / VR device 150 for review by the user and to an application using the input at 1150. At 1135, the user may select to correct the control input or text or may continue typing or generating another control motion. If the user continues typing or motioning at 1115, then at 1140 the user will initiate movement to enter text or provide the correct gesture, and the method loops at 1120 to continue motion detection. If the user selects to correct the input, such as by selecting all or some portion of the text output at 1135, then the user may correct the text or gesture at 1140 (by moving their fingers in a typing motion, or making a new gesture, as at 1115) and steps 1120, 1125, and 1130 are repeated resulting in a new control input. The corrected input is displayed to the display of the AR / VR device 150 at 1130 and an application on the AR / VR device which may be using the input at 1150.The method continues accepting input until the user gestures to stop tying input at 1145.

[0063] FIG. 12 illustrates a second method performed by a human input motion recognition technology application to implement the free-space human input motion recognition for text input. In embodiments, the method recognizes user typing motions in free space. FIG. 12 illustrates steps in addition to steps 1120 and 1125 (shown in FIG. 11 ) which are performed by the human input motion recognition application to respond to user input. At 1205, motion data is acquired by imaging sensor devices of the AR / VR device 150. In this embodiment, multiple imaging sensors such as RGB or RGBD (RGB depth) cameras provide a multi-sensor motion acquisition (in three dimensional space) over a time period. At 1210, hand pose estimation occurs as in previous embodiments. User profiles and preferences database 1215 can be used to store hand pose data for individual users, and as such the step of hand pose estimation at 1210 may comprise loading individual user pose data from database 1215. As in FIG. 5, user pose and keyboard mappings may include the x / y / z coordinate positions of the user’s hands and fingers relative to the keys associated with each finger, so that movement of the user’s finger to an x / y / z position (in conjunction with a detected event indicating the user intends to type or press that key, discussed below) generates an input for that letter. Such mapping occurs during the calibration process. As in previous embodiments, the user profiles and preferences 1215 may include keyboard or controller calibration data, and user preferences 1213 for keyboard layouts and feedback. At 1225 event detection of the user’s hand and fingers movement intended to create an input is evaluated. This includes typing motion and gestures enabling selection or letters, words, phrases or text, or gestures to start or stop a text entry operation. Event detection at 1225 may generate feedback 1220 The feedback at 1220 may be any of the types of feedback described herein and may be stored as a preference associated with the individual user at 1215. When a typing event is detected at 1225, text estimation at 1230 occurs. (Step 1230 is one embodiment of step 1125 of FIG> 11 .) Text estimation may occur as discussed above with respect to FIGs. 6 and 7 but adding a third physical dimension to the weighted fuse to determine whether a typing event has occurred.

[0064] In embodiments, step 1230 may occur using one of the techniques discussed below with respect to FIGs. 14A - 14C, 15 and 16. Text output 1130 is generated and may be provided to the display screen of the user device and to an application on the device 150 which uses text as input. At 1235 a determination is made as to whether a user-initiated correction event has occurred. If so, the method loops to step 1205 to receive additional input of the correction such as typing a new word and produces the corrected text at estimation step 1230. If no correction event occurs at 1235, at 1255, an automated modification event may occur. An automated or intelligent auto modification of the words or grammar of the text output may include a prompt by the system for with suggested corrections which may be selected by the user after the prompt, or an auto-correction of misspelled words or incorrect grammar at 1260.

[0065] FIG. 13 illustrates a second method performed by an application to implement the natural movement input technology for gesture input. At 1205, motion data is acquired by imaging sensors of the AR / VR device. In this embodiment, as in FIG. 12, multiple imaging sensors such as RGB or RGBD cameras provide a multisensor motion acquisition (in three dimensional space during a time period). At 1210, hand pose estimation occurs as in previous embodiments. User profiles and preferences database 1215 can be used to store hand pose data for individual users, and as such the step of hand pose estimation at 1210 may comprise loading individual user pose data from database 1215. Database 1315 may include gestures that include the x / y / z coordinate positions of the user’s hands and fingers relative positions in space (over time) which are mapped to specific control gestures, so that movement of the user’s hands and / or fingers in a specific pattern generates a control input associated with that gesture. At 1320 event detection detects movement of the user’s hand and fingers intended to create an input. In this example, this input includes a gesture having one or more defined movements which is understood by the system to produce an intended application input. Event detection at 1320 continues at 1325 until a goal position of the gesture is reached at 1325. If the gesture goal position is reached, at 1330, gesture classification determines which input is intended by the movements detected. If the goal position for the gesture is not reached, adetermination is made at 1335 as to whether feedback is needed to, for example, induce the user to make the correct gesture. At 1335, a determination is made as to whether feedback is required. If so, a feedback manager 1340 may be invoked which activates a feedback actuator 1345 to generate haptic feedback. The feedback manager 1240 step determines what type of feedback is needed. With a typing input, as in other embodiments, feedback may be more basic - a click or physical bump of the actuator to indicate a key stroke. For a gesture, more complex feedback such as an applied force to an actuator which varies in intensity depending on the nature of the gesture, may be useful. Once feedback is complete or if it is not needed, the method then loops to 1305 to continue looking for input. If not input is provided or if an end gesture is made at 1350, the method ends at 1355.

[0066] With reference to step 1230 and 1320 above, to determine the position of the fingers and hands for event detection, a third dimension is added to the method of FIG. 7. Three dimensions are used to track the palm and end landmarks of each finger (i.e. landmarks 4, 5 8, 9, 12, 13, 16, 17 and 20 for example). For each landmark, the X, Y and Z coordinates (relative to a defined coordinate system defined by the AR / VR device) are determined using the pose estimation over time. Again, one or more filtering techniques such as a moving average of the positions, a dilatated difference and / or a windowed standard deviation may be computed. A weighted fusing of the X, Y and Z coordinates occurs, followed by peak detection to determine when the user’s action indicates an attempt to “type” a key.

[0067] As noted herein, in embodiments, the human input motion recognition technology may be used with two dimensional RGB cameras. In other embodiments, stereo cameras may be used. A stereo camera is a type of camera with two or more lenses with a separate image sensor or film frame for each lens. This allows the camera to simulate human binocular vision, and therefore gives it the ability to capture three-dimensional images, a process known as stereo photography.

[0068] FIG. 14A illustrates a perspective and data use of stereo cameras in an AR / VR device to determine hand and finger positioning. A stereo camera observes the same scene from two or more viewpoints 01 and 02, as illustrated in FIG. 14A.Images are captured from slightly different perspectives using two parallel cameras (left and right) to calculate the parallax (horizontal shift) between corresponding pixels in these images. By comparing the positions of corresponding pixels in the left and right images the disparity (difference in position) between them is estimated. Closer objects have larger disparities, while distant ones have smaller disparities. To utilize a stereo camera in the present technology, the calibration steps above involve acquiring images in multiple planes (FIG. 14A) developing a set of rotation (R) and translation (T) matrices, computing the spatial relationship between the two cameras and defining a single rotation and translation matrix (R, T) that relates relative real-world coordinates for the left and right cameras. By computing the disparity for each pixel pair, a disparity map is created which encodes the depth information of the scene and allows estimation of the depth. The depth map provides information about how far each pixel is from the camera.

[0069] FIGs. 14B and 14C illustrate the calculated z-depth value relative to a horizontal X position of an index fingertip for a moving hand (FIG. 14B) and a finger “click” (FIG. 14C). respectively. For a moving hand, the X and Z positions track relatively well (as the whole hand is moving). However, in the “click” motion, the position of the index finger and Z depth disparity becomes more clear. In this instance, for example, a vertical click motion of the index finger, there is little to no movement of the finger in the x-plane, but substantial movement in the depth (Z) direction as one makes a click motion (pressing a button with their finger). The stereo camera detects this motion and can use the detection as a gesture template as described above.

[0070] FIGs. 15 and 16 illustrate another technique for determining position and movement of the human hand and fingers using machine learning. FIG 15 illustrates a set of inputs for each of the digits of a hand of, for example, the x-axis over time. FIG. 15 shows set of line graphs, each representing the change in wavelength data measured over time, with the horizontal-axis usually representing time and the verticalaxis representing the degree of change. There are five lines, each labeled with the name of a finger or part of the hand: Index, Middle, Wrist, Ring, and Little. The verticalaxis, labeled AA (nm) represents a change in wavelength in nanometers, representing optical measurement in the form of finger movement or flexion. Each line showsfluctuations in AA over a 60-second period. The patterns of fluctuation are distinct for each finger or part of the hand. The data produced and shown in FIG. 15 may be processed using the signal processing techniques of step 707 / 727 alone or in combination to achieve greater accuracy of the signal data.

[0071] FIG 16 illustrates a machine learning network which may be utilized to evaluate the data of FIG. 15 to determine a gesture or typing action. A machine learning network, such as illustrated in FIG. 16, can evaluate input signals (In), such as the x-axis, y-axis and z-axis positions of fingers over time, or the change of the input signal of such positions over time (as in FIG. 15) to determine the posture of a human hand. Positional data such as that illustrated in FIGs. 8 and 15 is gathered using imaging sensors through computer vision techniques tracking finger and hand movement. Relevant pose features may be utilized from pre-pose tracking techniques such as those described herein. For time-series data, such as the positions of a finger over time, recurrent neural networks (RNNs) may be used. These architectures are capable of capturing temporal dynamics to understand sequences of movements over time.

[0072] In FIG. 16, a classifier automated neural network (ANN) is shown which takes intensity values (FIG. 8) to the network as inputs and calculates a degree of positional matching to a target posture (ranging from 0 to 1 ) as the output. Any number of different types of ANNs may be used and can be implemented according to a feedforward backpropagation architecture. Subsequently, the output matching rates are submitted to a winner-take all competitive layer, returning the final classification estimated from the FMG input signals.

[0073] FIG. 17 illustrates a network processing device 1700 which may comprise an AR / VR processing device for implementing the technology discussed herein. The network processing device 1700 may be integrated into a headset 150a or may be provided in a housing which is separate from but connected to the headset 150a. In embodiments, network processing device 1700 may contain multiple instances of a component, such as multiple processing units, processors, memories, interfaces, etc.

[0074] Device 1700 may comprise a central processing unit (CPU) 1710, a graphics processing unit (GPU) 1720, a memory 1725, a mass storage device 1730, and an I / O interface 1715 connected to a bus 1770. The I / O interface 1715 may be connected to a headset 150a where the components of the network processing device are illustrated in FIG. 17 are not integrated with the headset 150a. The bus 1770 may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus, or the like. A network interface 1735 enables the network processing device to communicate over a network 1000 with other processing devices such as those described herein.

[0075] The mass storage device 1730 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus 1770. The mass storage device 1730 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like. The mass storage device 1730 includes instructions which when executed by the CPU (or processor) cause the processor to perform the methods described herein. The mass storage 1730 may include code in the form of application modules 1734 stored thereon, comprising instructions for causing the CPU to implement the components of natural movement application which are illustrated as present in memory 1725 in FIG. 17.

[0076] The CPU 1710 may comprise any type of electronic data processor. Memory 1725 may comprise any type of system memory such as static randomaccess memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, memory 1725 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 1725 is non-transitory. In one embodiment, the memory 1725 includes computer-readable instructions that are executed by the processor(s) 1710 and 1720 to implement embodiments of the disclosed technology.

[0077] In one embodiment, the memory 1725 includes instances of a data acquisition engine 1750, a keyboard layout selector 1752, a hand pose estimator1754, a keyboard layout calibration engine 1756, an event detector 1758, a feedback manager 1760, a text estimator 1762, an autocorrect module 1764 and an automodification module 1766. As noted above, stored versions of each of these components may be provided in mass storage 1730 in application modules 1734, although not illustrated in FIG. 17. Storage 1730 includes a user profiles and preferences database 1732. Database 1732 may store the information referenced above with respect to user preferences and profiles 508, 510 and 1215, for example.

[0078] The data acquisition engine 1750 comprises code that when executed on the processors implement steps 220, 502, 1205, and 1305, for example, detecting visual or depth sensor data indicating positions of a user’s hands and fingers. Keyboard selector 1752 comprises code that when executed on the processors implement steps 508 or selects one or more keyboard layout preferences from the user profile and preferences database 1732. Hand pose estimator 1754 comprises code that when executed on the processors implement steps 504 and 1210 for example. Keyboard layout calibration engine 1756 comprises code that when executed on the processors implement step 505 or retrieves a stored calibration for a user from user profile and preference database 1732. Event detector 1758 comprises code that when executed on the processors implement steps 514, 1225, and 1320, for example. Feedback manager 1760 comprises code that when executed on the processors implement steps 1340 or retrieves feedback preferences from the user profile and preference database 1732. Text estimator 1762 comprises code that when executed on the processors implement steps 425 and 1230, for example. Autocorrect module 1764 comprises code that when executed on the processors implement steps 522, 524, 1235, and 1240 for example. Auto modification module comprises code that when executed on the processors implement steps 528, 530, 1255 and 1260, for example.

[0079] The subject matter allows user typing to happen on any type of surfaces with any color or material. The hand pose estimation and typing event detection algorithms are not sensitive to the background. Dynamic and personalized keyboard layouts are utilized for improved user experiences. The virtual keyboard layout andsize maybe customized based on user preferences and profiles and may also be dynamically adjusted per user preferences or application and / or context.

[0080] As noted herein, haptic feedback is one type of feedback that may be utilized in conjunction with the human motion recognition technology. Haptic feedback engages users’ sense of control to enhance digital interactions. Haptic feedback validates user actions and interactions by providing tangible responses. In AR / VR space, when users press a button, swipe a screen, reach an object, or complete a gesture, haptic feedback can let user to adjust their actions, confirms their input, reducing uncertainty and contributing to a more intuitive sense of control and mastery. In embodiments herein, actuator 1775 may comprise a feedback actuator in the form of a separate user-wearable device such as a glove, rings, wristband, watch, tattoo or other clothing that provides force feedback to a user. In such embodiments, electrical power to such devices can be based on radio frequency energy harvesting, either from a central source or from a local, on-body source.

[0081] Energy harvesting refers to the ability to acquire operating energy from some sources of the operating environment. It could come from light, temperature differentials, vibration, RF waves, or many other physical phenomena. While some energy harvesting devices are implemented as separate units, they can also be combined with other common circuits. Renesas has a microcontroller (MCU) with energy-harvesting management capability. It can power the MCU as well as other circuits. Energy harvesting techniques can be used in conjunction with low-cost printed sensors to form an actuator 1775.

[0082] FIG. 18 illustrates one type of haptic feedback system. A glove 1800 includes an integrated energy harvester and controller 1810. A plurality of feedback sensors 1820, 1830, 1840, 1850 and 1860 are arranged on each finger of the glove, with each sensor comprising a flexible feedback actuator. The glove may be made of a flexible, durable, and lightweight material designed to fit snugly around the hand. Each sensor may comprise one or more multiple haptic actuators, such as piezoelectric actuators, to provide varying levels of vibration, pressure, or movementto simulate different tactile sensations. Additional sensors such as capacitive touch or resistive touch sensors may be placed on the glove.

[0083] FIG. 19 illustrates a block diagram of an energy harvester and controller 1810 which may be integrated with or attached to glove 1800. The energy harvester and controller 1810 is designed to capture and convert ambient RF signals into usable electrical energy. The energy harvester and controller 1810 includes an antenna 1905 which captures ambient RF signals 2000 from various sources. Any number of antenna types may be used. An energy harvester 1960 may harvest ambient RF energy 200 or energy based on movement of a user’s hand within the glove 1800. The harvester 1960 may include components such as a rectifier circuit to convert the captured RF signals into direct current (DC) voltage and a matching network which ensures power transfer from the antenna to a rectifier by matching the impedance of the antenna to the rectifier's input impedance. The energy harvester and controller 1810 may include one or more sensors 1910 such as gyroscopes, accelerometers, and compasses. The combination of sensors 1910 can detect the many subtle motions of a hand and can be used in combination with the imaging sensors when a glove 1800 is in communication with an ARA / R device implementing the described technology. A communication module 1930 is provided for wireless communication with an ARA / R device. A power management module 1940 regulates the harvested energy, ensuring a stable output voltage and managing the energy flow to the wearable device, and utilizes the energy harvester 1960 and power storage 1950 to supply harvested power to the sensors 1910, and 1820, 1830, 1840, 1850 and 1860. Storage 1950 comprises an energy storage component for use by the wearable device sensors and components of controller 1810. Controllers with energy harvesting capabilities such as controller 1810 are commercially available from a number of sources.

[0084] The disclosed technology provides for natural movement detection in an AR / VR device where computational resources and power consumption are at a premium. Typing input, for example, needs to be executed multiple times and yet maintain an acceptable user experience where latency is below a noticeable level toa user, environment, the computational resource and power consumption are usually constrained. The technology provides techniques to achieve these goals.

[0085] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the figures may not necessarily be drawn to scale.

[0086] For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe different embodiments or the same embodiment.

[0087] For the purposes of this document, a connection may be a direct connection or an indirect connection (e.g., via one or more other parts). In some cases, when an element is referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element via intervening elements. When an element is referred to as being directly connected to another element, then there are no intervening elements between the element and the other element. Two devices are “in communication” if they are directly or indirectly connected so that they can communicate electronic signals between them.

[0088] Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present disclosure.

[0089] The technology described herein can be implemented using hardware, software, or a combination of both hardware and software. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. The processor readable storage devices can include computer readable media such as volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computerreadable storage media and communication media. Computer readable storage media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by a computer. A computer readable medium or media does (do) not include propagated, modulated, or transitory signals.

[0090] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated, or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0091] In alternative embodiments, some or all of the software can be replaced by dedicated hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Applicationspecific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer readable media / storage devices, peripherals and / or communication interfaces.

[0092] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments setforth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.

[0093] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0094] The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0095] For purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices.Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.

[0096] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMSWhat is claimed is:

1. A computer implemented method of providing natural motion input to a computing device worn on a user’s head, comprising: defining one or more application control events each of which generate an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user; receiving imaging data over time from which movements and positioning of hands and fingers of the user can be determined; mapping multiple key points on the user’s hand and fingers; calculating whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; and for each motion and position of the user’s hands and fingers matching a defined application control event, outputting, to an application on the computing device, a corresponding application input.

2. The computer implemented method of claim 1 wherein the one or more application control events are defined by movement and positioning of hands and fingers relative to positions on a keyboard projected on a display of the computing device.

3. The computer implemented method of any of claims 1 through 2 wherein the control event is a keystroke entering a keyboard item.

4. The computer implemented method of any of claims 1 through 3 wherein the control event is a gesture selecting a word on the display of the computing device.

5. The computer implemented method of claim 3 wherein the keyboard items includes one or more letters and further including calculating a word based on the one or more letters.

6. The computer implemented method of claim 4 wherein the method further comprises repeating receiving and calculating to words selected by the gesture.

7. The computer implemented method of any of claims 1 through 6 wherein the calculating comprises: filtering at least two dimensions of the key points in the imaging data; and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions.

8. The computer implemented method of any of claims 1 through 7 wherein the filtering comprises one or more of: calculating a moving average to key point position coordinates positions calculated over a given time window; calculating a dilatated difference between the key point position coordinates; and calculating a windowed standard deviation of key point coordinates.

9. The computer implemented method of claim 8 wherein the key point coordinates are represented by two-dimensional coordinates.

10. The computer implemented method of claim 8 wherein the key point coordinates are represented by three-dimensional coordinates.

11. A user equipment device, comprising: a storage medium comprising computer instructions; a wearable display device; one or more processors coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the device to:define one or more application control events each of which generates an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user; receive imaging data over time from which movements and positioning of hands and fingers of the user can be determined; map multiple key points on the user’s hand and fingers; calculate whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; and for each motion and position of the user’s hands and fingers matching a defined application control event, output a corresponding application input to an application on the user equipment device.

12. The user equipment device of claim 11 wherein the one or more application control events are defined by movement and positioning of hands and fingers relative to positions on a keyboard projected on the display device.

13. The user equipment device of any of claims 11 through 12 wherein the control event is a keystroke entering a keyboard item.

14. The user equipment device of any of claims 11 through 13 wherein the control event is a gesture selecting a word on the display of the display device.

15. The user equipment device of any of claims 13 through 15 wherein the keyboard items includes one or more letters and further including calculating a word based on the one or more letters.

16. The user equipment device of any of claims 12 through 15 further comprising instructions which cause the device to repeat receiving and calculating to words selected by the gesture.

17. The user equipment device of any of claims 11 through 16 wherein the instructions cause the device to calculate by: filtering at least two dimensions of the key points in the imaging data; and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions.

18. The user equipment device of claim 17 wherein the filtering comprises one or more of: calculating a moving average to key point position coordinates positions calculated over a given time window; calculating a dilatated difference between the key point position coordinates; and calculating a windowed standard deviation of key point coordinates.

19. The user equipment device of claim 18 wherein the key point coordinates are represented by two-dimensional coordinates.

20. The user equipment device of claim 18 wherein the key point coordinates are represented by three-dimensional coordinates.

21. A non-transitory computer-readable medium storing computer instructions to recognize human motion input for an application, that configure at least one processor, upon execution of the instructions, to perform the following steps: defining one or more application control events each of which generate an application input, each control event being identified based on determined movement and positioning of hands and fingers of the user; receiving imaging data over time from which movements and positioning of hands and fingers of the user can be determined; mapping multiple key points on the user’s hand and fingers; calculating whether one or more control events occur in the imaging data based on locations of the key points in the imaging data over time in comparison to the movement and positioning defined for the one or more application control events; andfor each motion and position of the user’s hands and fingers matching a defined application control event, outputting, to an application on the computing device, a corresponding application input.

22. The non-transitory computer-readable medium of claim 21 wherein the one or more application control events are defined by movement and positioning of hands and fingers resembling input on a keyboard.

23. The non-transitory computer-readable medium of any of claims 21 through 22 wherein the control event is a keystroke entering a keyboard item.

24. The non-transitory computer-readable medium of any of claims 21 through 23 wherein the control event is a gesture selecting a word on a display of the computing device.

25. The non-transitory computer-readable medium of any of claims 23 through 24 wherein the keyboard items includes one or more letters and further including calculating a word based on the one or more letters.

26. The non-transitory computer-readable medium of any of claims 24 through 25 wherein the steps further comprise repeating receiving and calculating to words selected by the gesture.

27. The non-transitory computer-readable medium of any of claims 21 through 26 wherein the calculating comprises: filtering at least two dimensions of the key points in the imaging data; and determining an event by combining the at least two dimensions using a weighted fusing of the at least two dimensions.

28. The non-transitory computer-readable medium of claim 27 wherein the filtering comprises one or more of:calculating a moving average to key point position coordinates positions calculated over a given time window; calculating a dilatated difference between the key point position coordinates; and calculating a windowed standard deviation of key point coordinates.

29. The non-transitory computer-readable medium of claim 28 wherein the key point coordinates are represented by two-dimensional coordinates.

30. The non-transitory computer-readable medium of claim 28 wherein the key point coordinates are represented by three-dimensional coordinates.

Images