Gesture navigation

Abstract

A method performed by a head-mounted device includes recognizing a pinching gesture formed by a thumb and forefinger; based on recognizing the pinching gesture, presenting a graphical user interface; while presenting the graphical user interface, recognizing movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, selecting an action, the action being selected based on a direction of the recognized movement; and performing the action.

Description

GESTURE NAVIGATIONCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 625,897, filed on January 26, 2024, titled “GESTURE NAVIGATION,” which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Users can interact with a computing system, such as a head-mounted device, within a virtual reality, augmented reality, or extended reality environment, to provide instructions to the system.SUMMARY

[0003] A computing system, such as a head-mounted device, can present a graphical user interface to a user in response to recognizing a pinching gesture by the user. The graphical user interface can be presented within a virtual reality, augmented reality, or extended reality environment. The user can form the pinching gesture in the air in front of the user with a thumb and forefinger of a hand of the user. The user can move the thumb and forefinger and release the pinching gesture to select an icon. The system can respond to the selection of the icon by performing an action, such as launching an application or presenting icons associated with applications.

[0004] According to an example, a method performed by a head-mounted device includes recognizing a pinching gesture formed by a thumb and forefinger; based on recognizing the pinching gesture, presenting a graphical user interface; while presenting the graphical user interface, recognizing movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, selecting an action, the action being selected based on a direction of the recognized movement; and performing the action.

[0005] According to an example, a non-transitory computer-readable storage medium comprises instructions stored thereon. When executed by at least one processor, the instructions are configured to cause a computing system to recognize apinching gesture formed by a thumb and forefinger; based on recognizing the pinching gesture, present a graphical user interface; while presenting the graphical user interface, recognize movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, select an action, the action being selected based on a direction of the recognized movement; and perform the action.

[0006] According to an example, A computing system includes at least one processor and a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium includes instructions stored thereon that, when executed by the at least one processor, are configured to cause the computing system to recognize a pinching gesture formed by a thumb and forefinger; based on recognizing the pinching gesture, present a graphical user interface; while presenting the graphical user interface, recognize movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, select an action, the action being selected based on a direction of the recognized movement; and perform the action.

[0007] According to an example, a head-mounted device includes an image sensor configured to capture images of a hand, a display configured to present a graphical user interface, at least one processor, and a non-transitory' computer- readable storage medium comprising instructions stored thereon. When executed by the at least one processor, the instructions are configured to cause the head-mounted device to: recognize a pinching gesture formed by a thumb and forefinger based on a first set of images captured by the image sensor; based on recognizing the pinching gesture, present, via the display, a graphical user interface; while presenting the graphical user interface, recognize movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture based on a second set of images captured by the image sensor; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, select an action, the action being selected based on a direction of the recognized movement; and perform the action.

[0008] The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 shows a user wearing a head-mounted device and forming a gesture with a hand of the user, to which the head-mounted device responds by pressing a graphical user interface to the user.

[0010] FIGs. 2A through 2H show a hand forming an open gesture, forming a pinching gesture, moving the pinching gesture, and releasing the pinching gesture, and a computing system responding by generating a graphical user interface.

[0011] FIGs. 3A through 3C show a hand rotating and forming a pinching gesture and the computing system responding by presenting and ceasing to present a graphical user interface.

[0012] FIGs. 4A through 4E show a hand forming a pinching gesture, moving the pinching gesture, and releasing the pinching gesture, and the computing system responding by generating and ceasing to present a graphical user interface.

[0013] FIGs. 5A through 5C show a hand forming a pinching gesture and rotating, and the computing system responding by generating the graphical user interface.

[0014] FIGs. 6A through 61 show hand formations to which the computing system responds by generating a first portion of a graphical user interface. FIGs. 6F through 61 show hand formations to which the computing system responds by not generating any portion of a graphical user interface.

[0015] FIGs. 7A through 7C show hand movement to which the computing system responds by not generating a graphical user interface. FIG. 7D shows the hand remaining still, to which the computing system responds by generating a graphical user interface.

[0016] FIGs. 8A and 8B show a hand moving and the computing system maintaining a position of a graphical user interface with respect to the thumb and forefinger.

[0017] FIGs. 9A and 9B show an orientation of a head of a user changing with respect to a background object and a computing system maintaining an orientation of a graphical user interface with respect to the background object.

[0018] FIGs. 10A and 10C show a hand maintaining pinching for a time thatsatisfies a time threshold and a computing system responding by presenting information associated with an icon included in the graphical user interface.

[0019] FIGs. 11 A and 1 IB show a hand moving toward the user and the computing system keeping the graphical user interface between the hand and the face of the user.

[0020] FIGs. 12A through 121 show a hand forming a pinching gesture and a computing system responding by generating a diamond-shaped graphical user interface.

[0021] FIG. 12J shows a hand with another graphical user interface.

[0022] FIG. 13 shows a graphical user interface and icons shown in response to selection of an icon within the graphical user interface.

[0023] FIG. 14 is a block diagram of a computing system that can perform functions described herein.

[0024] FIGs. 15A, 15B, and 15C show an example of a head-mounted device.

[0025] FIG. 15D shows another implementation of a head-mounted device.

[0026] FIG. 16 is a flowchart showing a method performed by a computing system.

[0027] Like reference numbers refer to like elements.DETAILED DESCRIPTION

[0028] A user can interact with a computing system, such as a head-mounted device, that presents an extended reality environment (which can include a virtual reality environment or a augmented reality environment) to the user, with gestures. A technical problem with interfaces of extended reality environments is difficulty in navigating buttons or icons presented to a user. A technical solution to this technical problem is for the user to form a pinching gesture with a thumb and forefinger of a hand of the user, and the computing system to respond to the pinching gesture by presenting a graphical user interface to the user. The user can move the thumb and forefinger while maintaining the pinching gesture, and release the pinching gesture to select an icon included in the graphical user interface. The computing system can respond to the selection of the icon by performing an action associated with the icon. This technical solution has the technical benefit of accurate determinations of selections by the user.

[0029] In some implementations, the actions associated with icons included inthe graphical user interface can include navigating a menu. The menu can include icons associated with applications. The applications associated with the icons in the menu can include gaming applications, productivity applications, social networking applications, or applications for viewing media content, as non-limiting examples. In some implementations, the actions associated with the icons included in the graphical user interface includes presenting a home screen that is initially presented to the user, presenting an overview that shows more applications than the home screen, and returning to a previous view. In some implementations, the graphical user interface includes a home button or home icon associated with the action of presenting the home screen. In some implementations, the graphical user interface includes an overview button or home icon associated with the action of presenting the overview. In some implementations, the graphical user interface includes a back button or back icon associated with the action of returning to the previous view.

[0030] FIG. 1 shows a user 102 wearing ahead-mounted device 104 and forming a gesture with a hand 106 of the user 102, to which the head-mounted device 104 responds by presenting a graphical user interface 108 to the user 102. The headmounted device 104 is mounted on a head of the user 102. In some implementations, the head-mounted device 104 includes smartglasses that project images onto lenses of the smartglasses to generate an overlay of images over the physical environment beyond the smartglasses. In some implementations, the head-mounted device 104 includes a headset that encloses eyes of the user 102 and includes a left display for a left eye of the user 102 and a right display for a right eye of the user 102 to present an immersive environment for the user 102.

[0031] The head-mounted device 104 includes at least one camera 110 facing away from eyes of the user 102. The camera 110 facing away from the eyes user 102 can capture images of objects in front of the user 102 as well as images of a hand 106 (or both hands) of the user 102. The head-mounted device 104 can recognize gestures formed by the hand 106 of the user 102 based on images captured by the camera 110. The head-mounted device 104 can perform actions based on the gestures recognized by the head-mounted device 104.

[0032] In some implementations, the head-mounted device 104 recognizes the hand 106 forming a pinching gesture. Based on recognizing the pinching gesture, the head-mounted device 104 can generate a graphical user interface 108. The head-mounted device 104 can generate the graphical user interface 108 by presenting and / or displaying the graphical user interface 108 on a display included in the headmounted device 104. The graphical user interface 108 is shown in broken lines in FIG. 1 to indicate the location at which the graphical user interface 108 appears to the user 102, while the graphical user interface 108 is actually displayed on one or more lenses of the head-mounted device 104 or by one or more displays included in the head-mounted device 104. In some implementations, the graphical user interface 108 generated by the computing system is a portion (such as a first portion 218) of a full graphical user interface that the computing system will generate in response to the user 102 forming a pinching gesture with the hand 106.

[0033] While the recognition of the gesture and generation of the graphical user interface 108 is described with respect to the head-mounted device 104, this is an example implementation. The methods, functions, and / or techniques described herein can be performed by other computing systems, such as a desktop or laptop computer with a camera (or other sensing device or image sensor) that recognizes gestures and performs actions based on the gestures.

[0034] FIGs. 2A through 2H show the hand 106 forming an open gesture, forming a pinching gesture, moving the pinching gesture, and releasing the pinching gesture, and a computing system responding by generating a graphical user interface 208. The hand 106 can be the hand 106 of the user 102 shown and described with respect to FIG. 1. The graphical user interface 208 (shown in FIGs. 2D, 2E, 2F, 2G, and 2H) can be an example of the graphical user interface 108 show n and described with respect to FIG. 1. The head-mounted device 104 is an example of the computing system that can recognize the pinching gesture and generate the graphical user interface 208.

[0035] FIG. 2A shows the hand 106 forming an open gesture. The open gesture be a formation of the hand 106 that includes a predefined orientation of a thumb 212 included in the hand 106 with respect to a forefinger 214 included in the hand 106. In some implementations, the computing system (such as the head-mounted device 104) can recognize the open gesture based on the orientation of the thumb 212 with respect to the forefinger 214 falling within a predetermined range, and / or an angle 0 formed by an intersection between the thumb 212 and forefinger 214 falling within the predetermined range. In some implementations, the predetermined range isfrom thirty degrees (30°) and sixty degrees (60°). In some implementations, the computing system recognizes the open gesture based on recognizing the thumb 212 as being in a straight position or extended position and recognizing the forefinger 214 as being in a straight position or extended position. The computing system can recognize the straight position or extended position of the thumb 212 or forefinger 214 based on angles of joints within the thumb 212 or forefinger 214. In some implementations, the computing system can recognize the forefinger 214 as being in the straight position or extended position based on each of the two joints within the forefinger 214 having angles of less than a predetermined value, such as fifteen degrees (15°), from horizontal. In some implementations, the computing system can recognize the thumb 212 as being in the straight position or extended position based on an angle of the joint within the thumb 212 indicating that the thumb 212 is either straight or pointing away from the forefinger 214 but not toward the forefinger 214, so that a doublejointed thumb is considered to be in the straight position or extended position when the thumb 212 is pointing away from the forefinger 214.

[0036] The computing system can respond to recognition of the open gesture by presenting a first portion 218 of the graphical user interface 208 (the full graphical user interface 208 is shown in FIGs. 2D and 2E). The computing system can recognize the open gesture and / or present the first portion 218 after the open gesture has been maintained for a predetermined open delay time. The computing system can determine that the open gesture was maintained for the predetermined open delay time by determining a time since an initial image indicated formation of the open gesture and a current time when the open gesture was maintained. The computing system can recognize the open gesture based on recognizing the open gesture and determining that the open gesture was maintained for the open delay time. The open delay time can prevent false triggers of the open gesture and / or first portion 218.

[0037] FIG. 2B shows the computing system presenting the first portion 218 of the graphical user interface 208. In the example shown in FIG. 2B. the first portion 218 is a central portion of the graphical user interface 208. The presentation of the first portion 218 by the computing system can indicate to the user 102 that the computing system can generate the graphical user interface 208 and / or that the user 102 can interact with the graphical user interface 208. In some implementations, the computing system presents the first portion 218 in a fixed position with respect to thehand 106, thumb 212, and / or forefinger 214 while the user 102 moves the hand 106. Maintaining the first portion 218 in the fixed position with respect to the hand 106, thumb 212, and / or forefinger 214 while the user 102 moves the hand 106 can enable to user 102 to form a pinching gesture at a location overlaid by the first portion 218.

[0038] FIG. 2C shows the hand 106 forming a pinching gesture. The hand 106 can form the pinching gesture by curling the thumb 212 and forefinger 214 toward each other so that an end portion of the thumb 212 contacts an end portion of the forefinger 214. In some implementations, the computing system recognizes the pinching gesture based on recognizing contact between the end portion of the thumb 212 and the end portion of the forefinger 214. In some implementations, the computing system recognizes the pinching gesture based on recognizing contact between the end portion of the thumb 212 and the end portion of the forefinger 214 at a location overlaid by the first portion 218 of the graphical user interface 208. The computing system can recognize the pinching gesture based on recognizing the contact between the end portion of the thumb 212 and the end portion of the forefinger within a first set of images captured by the camera 110. The first set of images can include one or multiple images captured by the camera 110. In some implementations, the computing system moves the first portion 218 of the graphical user interface 208 to a location overlaying the location at which the end portion of the thumb 212 contacts the end portion of the forefinger 214 to create a visually appealing interaction between the pinching gesture formed by the hand 106 of the user 102 and the first portion 218 of the graphical user interface 208.

[0039] FIG. 2D shows the graphical user interface 208 presented by the computing system. The computing system presents the graphical user interface 208, and / or presents a remainder, additional portion, and / or remaining portions, of the graphical user interface 208 (in addition to the first portion 218), based on and / or in response to recognizing the pinching gesture. The computing system can recognize the pinching gesture and / or present the graphical user interface 208 after the pinching gesture has been maintained for a predetermined pinching delay time. The pinching delay time can prevent false triggers of the pinching gesture and / or graphical user interface 208. In some implementations, the appearance of the first portion 218 changes when the computing system presents the graphical user interface 208 and / or remaining portions of the graphical user interface 208. In the example shown in FIG.2D. the graphical user interface 208 is a horizontal interface with three buttons or icons arranged in a horizontal row. The computing system can maintain the presentation of the graphical user interface 208 while the user 102 maintains the pinching gesture.

[0040] In the example shown on FIG. 2D, the graphical user interface 208 includes a home button or home icon 222 (which can be considered a first button or first icon) associated with an action of presenting a home screen that is initially presented to the user 102. The home icon 222 can also be considered a launcher icon or launcher button. The home icon 222 can be the first portion 218 of the graphical user interface 208 shown in FIGs. 2B and 2C. In the example shown on FIG. 2D. the graphical user interface 208 includes a back button or back icon 224 (which can be considered a second button or second icon) associated with an action of returning to a previous view. In the example shown on FIG. 2D, the graphical user interface 208 includes an overview button or overview icon 226 (which can be considered a third button or third icon) associated with an action of presenting an overview that shows more applications than the home screen.

[0041] In some implementations in which the graphical user interface 208 is a horizontal interface, the computing system presents the graphical user interface 208 in a fixed position with respect to objects in a background and / or within a physical environment beyond the hand 106. In the example shown in FIG. 2D, a dresser 220 is an object in the background and / or physical environment. In some implementations, the computing system presents the graphical user interface 208 in a fixed horizontal position with respect to background objects. In some implementations, the computing system moves the graphical user interface 208 vertically to correspond to vertical movements of the hand 106. Presenting the graphical user interface 208 in the fixed position or fixed horizontal position enables the user 102 to move the pinching gesture horizontally along the graphical user interface 208 to select buttons and / or icons within the graphical user interface 208.

[0042] FIG. 2E show s the graphical user interface 208 presented by the computing system after the user 102 has moved the hand 106 to the left while maintaining the pinching gesture. In the example shown in FIG. 2E, the user 102 has moved the hand 106 to the left and beyond the graphical user interface 208. The computing system recognizes the movement of the hand 106 in a leftward (or firsthorizontal) direction. The computing system can recognize the movement of the hand 106 in the leftward (or first horizontal) direction from the home icon 222 to the back icon 224 based on movement of the hand 106 in the leftward (or first horizontal) direction satisfying a distance threshold. The computing system can recognize the movement of the hand 106 based on a second set of images captured by the camera 110. The second set of images captured by the camera 110 can include multiple images, such as a first image of the hand 106 in the location shown in FIG. 2D and a final or last image of the hand in the location show n in FIG. 2E. In some examples, the second set of images can include intermediate images showing the hand 106 in locations between the location of the hand 106 in FIG. 2D and the location of the hand 106 in FIG. 2E. The computing system can associate the pinching gesture with the back icon 224 (or second button or second icon) based on recognizing movement of the hand 106 in the leftw ard (or first horizontal) direction. The circle around the back icon 224 indicates a preselection of the back icon 224. The computing system will recognize selection of the preselected back icon 224 in response to release of the pinching gesture.

[0043] FIG. 2F shows a second portion 228 of the graphical user interface 208 after the hand 106 of the user 102 has released the pinching gesture. The computing system can recognize the release of the pinching gesture after the movement of the hand 106 in the leftward (or first horizontal) direction as a selection of the back icon 224 (the back icon 224 is labeled in FIGs. 2D and 2E, and the back icon 224 can be considered to be the second portion 228 of the graphical user interface 208). The computing system can perform an action associated with the back icon 224 based on the selection of the back icon 224, such as returning to a previous screen. The computing system can present a second portion 228 of the graphical user interface 208, and / or cease presenting portions of the graphical user interface 208 other than the second portion 228, based on the release of the pinching gesture and / or selection of the back icon 224 by the hand 106. The second portion 228 that the computing system continues to present can be the back icon 224.

[0044] FIG. 2G show s the hand 106 w ithout any portion of the graphical user interface 208 presented. The computing system can cease presenting any portion of the graphical user interface 208 after a predetermined time duration has passed since the computing system began presenting the second portion 228 and / or ceasedpresenting portions of the graphical user interface 208 other than the second portion 228. The ceasing of presenting the graphical user interface 208 can free the user 102 to perform other tasks or prompt the computing system to generate and / or present the graphical user interface 208 in response to the user 102 forming the pinching gesture again.

[0045] FIG. 2H shows the graphical user interface 208 presented by the computing system after the user 102 has moved the hand 106 to the right while maintaining the pinching gesture. FIG. 2H is a scenario taking place after the scenario of FIG. 2D, in which the user 102 moved the hand 106 to the right instead of to the left. In the example shown in FIG. 2H, the user 102 has moved the hand 106 to the right and beyond the graphical user interface 208. The computing system recognizes the movement of the hand 106 in a rightward (or second horizontal) direction. The computing system can recognize the movement of the hand 106 in the rightward (or second horizontal) direction from the home icon 222 to the overview icon 226 based on movement of the hand 106 in the rightward (or second horizontal) direction satisfying a distance threshold. The computing system can associate the pinching gesture with the overview icon 226 (or third button or third icon) based on recognizing movement of the hand 106 in the rightward (or second horizontal) direction. The circle around the overview icon 226 indicates preselection of the overview icon 226. The computing system will recognize selection of the preselected overview icon 226 after the pinching gesture is released.

[0046] In response to the user 102 releasing the pinching gesture, the computing system can present a third portion of the graphical user interface 208 for a predetermined time and then no portion of the graphical user interface 208, in a similar manner as described above with respect to FIGs. 2F and 2G. The third portion of the graphical user interface 208 can be the overview icon 226. The computing system can recognize the release of the pinching gesture after movement of the hand 106 in the rightward (or second horizontal) direction as a selection of the overview icon 226. The computing system can respond to the selection of the ovendew- icon 226 by performing an action associated with the overview icon 226 such as presenting an overview that shows more applications than the home screen.

[0047] FIGs. 3 A through 3C show the hand 106 rotating and forming a pinching gesture and the computing system responding by presenting and ceasing topresent a graphical user interface. In the example shown in FIG. 3 A, the hand 106 is rotated away from the camera 110 included in the computing system that captures images of the hand 106. The orientation of the hand 106 rotated away from the camera 110 and / or user 102 can be considered a palm out position. Based on the hand 106 being rotated away from the camera 110, the computing system does not generate a graphical user interface. In some implementations, the computing system generates the graphical user interface based on the hand 106 forming the open gesture and an orientation of the hand 106 indicating that the hand 106 is facing the camera. In some implementations, the computing system can determine that the hand 106 is facing the camera 110 and / or user 102 based on at least a portion of a palm 306 (labeled in FIG. 3B) being visible to the camera 110. In some implementations, the computing system can determine that the hand 106 is facing the camera 110 based on at least a predetermined portion or proportion of the palm 306 being visible to the camera 110.

[0048] FIG. 3B shows the first portion 218 of the graphical user interface after the user 102 has rotated the hand 106 so that the palm 306 of the hand 106 is facing the camera 110, formed the open gesture, and begun forming a pinching gesture. The orientation of the hand 106 rotated away from the camera 110 and / or user 102 can be considered a palm in position. The computing system responded to the rotation of the hand 106 so that the hand 106 faces the camera 110 and the formation of the open gesture by presenting the first portion 218.

[0049] FIG. 3C shows the second portion 228 of the graphical user interface after the user 102 has formed the pinching gesture, rotated the hand 106 in the leftward or second direction, and released the pinching gesture. While not shown in FIGs 3B or 3C, the computing system has responded to the pinching gesture bypresenting the graphical user interface 208. The user 102 can quickly rotate the hand 106 to select the back icon 224 or second portion 228 without waiting for the computing system to present the graphical user interface 208.

[0050] The computing system has responded to the rotation of the hand 106 in the leftward or second direction and subsequent release of the pinching gesture by recognizing a selection of the back icon 224. The computing system responds to the selection of the back icon 224 by performing an action associated with the back icon 224, such as presenting a previous screen or set of icons. The computing system responds to the selection of the back icon 224 by ceasing presentation of portions ofthe graphical user interface 208 other than the back icon 224. The computing system can cease presentation of the back icon 224 a predetermined time after ceasing presentation of portions of the graphical user interface 208 other than the back icon 224 (or second portion 228). The computing system can, for example, fade the back icon 224 from view, increasing the transparency of the back icon 224 until the back icon 224 is no longer visible and / or presented.

[0051] FIGs. 4A through 4E show the hand 106 forming a pinching gesture, moving the pinching gesture, and releasing the pinching gesture, and the computing system responding by generating and ceasing to present the graphical user interface 208.

[0052] FIG. 4 A shows the hand 106 forming the open gesture. The computing system recognizes the open gesture and responds by presenting the first portion 218 of the graphical user interface 208.

[0053] FIG. 4B shows the hand 106 forming the pinching gesture and rotating the hand 106 in the leftward or first direction. The computing system recognizes the pinching gesture and responds to the pinching gesture by generating and / or presenting the full graphical user interface 208. FIG. 4C shows the hand 106 maintaining the pinching gesture and continuing to rotate the hand 106 in the leftward or first direction.

[0054] FIG. 4D shows the hand 106 after rotation in the leftward or first direction and release of the pinching gesture. The computing system can recognize the release of the pinching gesture after rotation of the hand 106 in the leftward or first direction as a selection of the back icon 224. The computing system can respond to the selection of the back icon 224 by performing an action, such as displaying a previous screen or previous set of icons. The computing system can respond to the selection of the back icon 224 by ceasing presentation of portions of the graphical user interface 208 other than the back icon 224 (which can also be considered the second portion 228). FIG. 4D shows the back icon 224 without other portions of the graphical user interface 208.

[0055] After ceasing presentation of portions of the graphical user interface 208 other than the back icon 224, the computing system can cease presentation of the back icon 224. The computing system can cease presentation of the back icon 224 a predetermined time, such as one second, after ceasing presentation of portions of thegraphical user interface 208 other than the back icon 224. FIG. 4E shows the hand 106 after the computing system has ceased presentation of any portion of the graphical user interface 208.

[0056] FIGs. 5A through 5C show a hand 106 forming a pinching gesture and rotating, and the computing system responding by generating the graphical user interface 208. FIG. 5 A shows the hand 106 forming the pinching gesture with the palm 306 visible to the camera 110. The computing system presented the graphical user interface 208 based on recognizing the pinching gesture while the palm 306 was visible to the camera 110.

[0057] FIG. 5B shows the hand 106 rotated in the leftward or first direction while maintaining the pinching gesture. The computing system recognizes the pinching gesture as being maintained based on the end portion of the thumb 212 maintaining contact with the end portion of the forefinger 214. Based on the pinching gesture being maintained, the computing system does not recognize the selection of an icon because the pinching gesture has not been released. Based on the pinching gesture being maintained, the computing system continues to present the graphical user interface 208. The circle around the back icon 224 indicates preselection of the back icon 224 based on the movement in the leftward direction or first direction and the pinching gesture being maintained. The circle is an example of a visual indication of a preselection.

[0058] FIG. 5C shows the hand 106 rotated further in the leftward or first direction while maintaining the pinching gesture. The rotation of the hand 106 is such that the none of the palm 306 of the hand 106 is visible to the camera 110 capturing images of the hand 106. The rotation of the hand 106 is such that the palm 306 of the hand 106 can be considered to be facing away from the camera 110 and / or computing system. While the computing system would not have initially presented the graphical user interface 208 if the hand 106 formed the pinching gesture while the hand 106 was facing away from the camera 110. the computing system continues to present the graphical user interface 208 while the pinching gesture is maintained with the hand 106 facing away from the camera based on the formation of the pinching gesture while the palm 306 of the hand 106 was facing the camera 110. Based on the pinching gesture being maintained, the computing system does not recognize the selection of an icon because the pinching gesture has not been released. 208.

[0059] FIGs. 6A through 61 show hand formations to which the computing system responds by generating the first portion 218 of the graphical user interface 208. FIGs. 6F through 61 show hand formations to which the computing system responds by not generating any portion of the graphical user interface. As shown in FIGs. 6A through 6E, the thumb 212 and forefinger 214 form the open gesture. The computing system recognizes the hand 106 as forming the open gesture based on the thumb 212 and forefinger 214 forming the open gesture. The computing system recognizes the hand 106 as forming the open gesture based on the thumb 212 and forefinger 214 forming the open gesture despite different positions of fingers 606 other than the thumb 212 and forefinger 214. Based on the hand 106 forming the open gesture, the computing system presents the first portion 218. The representation of the first portion 218 as a diamond inside of a diamond in FIG. 6A and subsequent figures is an alternative representation of the first portion 218 compared to the representation of the first portion 218 as a circle inside of a diamond shown in FIGs. 1 through 4A.

[0060] As shown in FIGs. 6F through 61, the thumb 212 and forefinger 214 do not form the open gesture. The remaining fingers 606 are in various positions. Based on the thumb 212 and forefinger 214 not forming the open gesture, the computing system does not recognize an open gesture and does not present the first portion 218 or any other portion of the graphical user interface 208. The computing system recognizes the open gesture, or lack of open gesture, based on the orientation of the thumb 212 and forefinger 214 independently of orientations and / or positions of fingers 606 other than the thumb 212 and forefinger 214.

[0061] FIGs. 7A through 7C show hand movement to which the computing system responds by not generating a graphical user interface. FIG. 7D shows the hand 106 remaining still, to which the computing system responds by generating a graphical user interface. In some implementations, while the hand 106 is moving, the computing system will not generate or initiate presentation of any portion of the graphical user interface 208. In some implementations, the computing system will generate and / or initiate presentation of the graphical user interface (or a portion thereof) based on the hand 106 forming the open gesture and the hand 106 being in a stationary position. In some implementations, the computing system will generate and / or initiate presentation of the graphical user interface (or a portion thereof) based on the hand 106 forming the open gesture, the palm 306 facing the camera, and thehand 106 being in a stationary position.

[0062] FIGs. 7A through 7C show the hand 106 moving to the left and right. The hand 106 is moving with respect to a background object. In the examples shown in FIGs. 7A through 7D, the background object is the dresser 220. Even though the hand 106 is forming the open gesture and the palm 306 is facing the camera 110, the computing system does not generate or initiate presentation of any portion of the graphical user interface 208 while the hand is moving.

[0063] FIG. 7D shows the hand 106 after remaining in a stationary position that the hand 106 arrived at in FIG. 7C. Based on the hand 106 forming the open gesture and being in the stationary position, the computing system generates and / or initiates presentation of the first portion 218. The computing system can determine that the hand 106 is stationary based on movement over a predetermined time period, which can be considered a movement time duration, being less than a movement threshold. The computing system may, for example, compare sequential images of the hand 106 during sliding windows of the predetermined time period to determine whether movement between images exceeds the movement threshold. If the movement does not exceed the movement threshold, then the computing system can determine that the hand 106 is stationary.

[0064] FIGs. 8A and 8B show the hand 106 moving and the computing system maintaining a position of the graphical user interface 208 with respect to the thumb 212 and forefinger 214. The computing system has generated the first portion 218 of the graphical user interface 208 based on the hand 106 forming the open position. The hand 106 moves back and forth. The hand 106 can move with respect to a background object such as the dresser 220. The computing system can present the first portion 218, or the full graphical user interface 208, in a fixed position with respect to the thumb 212 and forefinger 214 while the hand 106 and / or thumb 212 and forefinger 214 move with respect to the background object. The computing system can move the first portion 218 and / or graphical user interface 208 with respect to the background object so that movement of the first portion 218 and / or graphical user interface 208 corresponds to movement of the hand 106 and / or thumb 212 and forefinger 214.

[0065] FIGs. 9A and 9B show an orientation of a head of the user 102 changing with respect to a background object and a computing system maintaining an orientation of the graphical user interface 208 with respect to the background object.While the head of the user 102 is not shown, FIG. 9B shows a different perspective of the same scene than FIG. 9Ato reflect the changed orientation of the head of the user 102 and / or the camera 110. The computing system can maintain an orientation of the graphical user interface 208 with respect to a background object, such as the dresser 220, while an orientation of a head of the user 102 changes with respect to the background object. The position of the graphical user interface 208 remains constant with respect to the hand 106. The change of the orientation of the head of the user 102 can cause an orientation of a camera included in the computing system (such as the head-mounted device 104) to change with respect to background objects. To maintain continuity of appearance of the graphical user interface 208 with respect to the background object, the computing system can rotate the presentation of the graphical user interface 208 within the display.

[0066] FIG. 9A shows the graphical user interface 208 after the user 102 has formed the pinching gesture and the computing system has initiated presentation of the graphical user interface 208. FIG. 9B shows the graphical user interface 208 after the user 102 has rotated a head and / or body of the user 102 from the position of FIG. 9A, causing the position and / or orientation of the camera 110 to change with respect to the dresser 220, while maintaining the pinching gesture. The computing system rotates the appearance of the graphical user interface 208 to maintain the orientation of the graphical user interface 208 with respect to the background object, of which the dresser 220 is the example background object in the example of FIG. 9B.

[0067] FIGs. 10A through 10C show a hand 106 maintaining pinching for a time that satisfies a time threshold and a computing system responding by presenting information 1006 associated with an icon included in the graphical user interface 208. Satisfaction of the time threshold can also be considered satisfaction of a pinching time condition based on maintaining the pinching gesture for at least a pinching time threshold. The computing system can determine whether the hand 106 has remained in a stationary position while maintaining the pinching gesture for a time that satisfies an information time threshold. The information time threshold can be longer than the movement time duration. The computing system can determine whether the hand 106 has remained in the stationary position, and / or how long the hand 106 remained in the stationary’ position, in a similar manner as described above with respect to FIGs. 7A through 7C. The computing system can present the information based on the hand 106maintaining the pinching gesture and remaining stationary for the time that satisfies the information time threshold.

[0068] FIG. 10A shows initiation of the pinching gesture after the computing system has presented the first portion 218 of the graphical user interface 208 based on the hand 106 forming the open gesture. FIG. 10B shows the computing system presenting the full graphical user interface 208 based on the hand 106 forming the pinching gesture. The computing system presents the circle around the home icon 222 to indicate preselection of the home icon 222 based on the hand 106 maintaining the pinching gesture and remaining stationary for a time that satisfies a preselection time threshold. FIG. 10C shows the computing system presenting the information 1006 based on the hand 106 maintaining the pinching gesture and remaining stationary for the time that satisfies the information time threshold. The computing system can begin a timer to determine whether the hand 106 maintained the pinching gesture and remained stationary for the time that satisfies the information time threshold after initiating presentation of the full graphical user interface 208 and / or after presenting the circle around the home icon 222. The information 1006 can include information about the home icon 222, such as an action that will occur if the user 102 selects the home icon 222.

[0069] FIGs. 11A and 1 IB show a hand 106 moving toward the user 102 and the computing system keeping the graphical user interface 208 between the hand 106 and the face of the user 102. FIG. 11B shows the hand 106 closer to the face of the user 102 and / or camera 110 included in the computing system than in FIG. 11 A. The computing system will maintain a position of the graphical user interface 208 while the hand 106 is moving. However, the computing system will keep the graphical user interface 208 between the hand 106 and the face of the user 102, preventing the hand 106 from occluding the view of the graphical user interface 208. If the hand 106 moves to a location that would be between the graphical user interface 208 and the face of the user 102, the computing system can move the graphical user interface 208 toward the face of the user 102 so that the graphical user interface 208 remains between the hand 106 and the face of the user 102.

[0070] FIGs. 12A through 121 show a hand 106 forming a pinching gesture and a computing system responding by generating a diamond-shaped graphical user interface 1208. In this implementation, the graphical user interface 1208 generated bythe computing system in response to the computing system recognizing the pinching gesture has a shape of a diamond rather than a shape of a horizontal bar as in the examples shown in FIGs. 2D, 2E, 2H, 4B, 4C, 5A through 5C, 9A, 9B, 10B, IOC, 11 A, and 11B. In this implementation, the graphical user interface 1208 includes four icons. Other features and functionalities of the graphical user interface 1208 are similar to those described above with respect to the graphical user interface in a shape of a horizontal bar.

[0071] The computing system can generate a portion 1218 of the graphical user interface 1208 by recognizing the hand 106 forming an open gesture in a similar manner as described above with respect to the first portion 218. FIG. 12A shows the computing system presenting the portion 1218 of the graphical user interface 1208 in response to the hand 106 forming the open gesture. FIG. 12B shows the hand 106 forming the pinching gesture after the computing system generated and / or presented the portion 1218 of the graphical user interface 1208. The computing system can recognize the pinching gesture shown in FIG. 12B.

[0072] FIG. 12C shows the diamond-shaped graphical user interface 1208 generated and / or presented by the computing system. The computing system generated and / or presented the diamond-shaped graphical user interface 1208 based on recognizing that the hand 106 formed the pinching gesture.

[0073] In the implementation shown in FIG. 12C, the graphical user interface 1208 includes four icons. The four icons are arranged directionally, with each of the four icons corresponding to a different direction from a center of the graphical user interface 1208. In an example implementation, the four icons include first icon or home icon 1222 in a first direction or an upward direction from a center of the graphical user interface 1208, a second icon or back icon 1224 in a second direction or leftw ard direction from the center of the graphical user interface 1208, a third icon or overview icon 1226 in a third direction or rightward direction from the center of the graphical user interface 1208, and a fourth icon or taskbar icon 1228 in a fourth direction or downward direction from the graphical user interface 1208. The home icon 1222 can be associated with a similar action as the home icon 222 described above. The back icon 1224 can be associated with a similar action as the back icon 224 described above. The overview icon 1226 can be associated with a similar action as the overview icon 226. The taskbar icon 1228 can be associated with an action ofpresenting a taskbar with applications to launch.

[0074] FIG. 12D shows the hand 106 after the hand 106 has moved in the first direction or upward direction toward the home icon 1222. The circle around the home icon 1222 indicates preselection of the home icon 1222. FIG. 12E shows the hand 106 after the hand 106 has moved in the third direction or rightward direction tow ard the overview icon 1226. The circle around the overview icon 1226 indicates preselection of the overview icon 1226. FIG. 12F shows the hand 106 after the hand 106 has moved in the fourth direction or downward direction toward the taskbar icon 1228. The circle around the taskbar icon 1228 indicates preselection of the taskbar icon 1228.

[0075] FIG. 12G shows the hand 106 after the hand 106 has moved in the second direction or leftward direction tow ard the back icon 1224. The circle around the back icon 1224 indicates preselection of the back icon 1224. FIG. 12H show s the hand 106 beginning to release the pinching gesture while the pinching gesture is over the home icon 1222. The computing system can respond to the release of the pinching gesture w hile the pinching gesture is over the home icon 1222 by performing the action associated with the home icon 1222. The computing system can also respond to the release of the pinching gesture while the pinching gesture is over the home icon 1222 by ceasing presentation of the graphical user interface 1208. FIG. 121 shows the computing system fading the graphical user interface 1208 out by increasing a transparency of the graphical user interface 1208. The graphical user interface 1208 may thereafter not be visible and / or presented.

[0076] FIG. 12J shows a hand 106 with another graphical user interface 1258. In this example, the graphical user interface 1258 includes five buttons or icons 1272, 1274, 1276, 1278, 1280. The computing system can generate the graphical user interface 1258 based on a palm in gesture, open gesture, and / or pinching gesture as described above with respect to the graphical user interface 108, graphical user interface 208, and / or graphical user interface 1208. The user 102 can select the 1272. 1274, 1276, 1278, 1280 in a similar manner to selecting the icons 222, 224, 226, 1222, 1224, 1226, 1228 described above, such as by moving the pinching gesture in a direction of an icon and releasing the pinching gesture or releasing the pinching gesture without moving the pinching gesture in the example of a center icon or fifth icon 1280. The icons 1272, 1274, 1276, 1278, 1280 can be associated with similaractions as the 222, 224, 226, 1222. 1224, 1226, 1228 described above, or can be customized to be associated with actions selected by a user or administrator, as nonlimiting examples.

[0077]

[0078] FIG. 13 shows a graphical user interface 1308 and icons 1302 shown in response to selection of an icon within the graphical user interface 1308. The selected icon can be a home icon 1222. The computing system can present the icons 1302 in response to the hand pinching and releasing near the home icon 1222. The icons 1302 can be included in a menu or home screen, or include an overview of applications.

[0079] FIG. 14 is a block diagram of a computing system 1400 that can perform functions described herein. The computing system 1400 can include a headmounted device such as the head-mounted device 104, a computing device in communication with the head-mounted device 104, a desktop or laptop computer, or any other computing device.

[0080] The computing system 1400 can include an environmental monitor 1402. The environmental monitor 1402 can monitor the environment, such as a visual environment, surrounding the computing system 1400. The environmental monitor 1402 can monitor the environment based on images captured by a camera, such as the camera 110, included in or in communication with the computing system 1400. The environmental monitor 1402 can process images captured by the camera 110. The environmental monitor 1402 can, for example, determine whether the environment has changed based on differences between images captured by the camera. The environmental monitor 1402 can, for example, determine whether a position of a hand such as the hand 106 has changed and / or whether a shape or form of the hand has changed. The environmental monitor 1402 can prompt a gesture recognizer 1404 included in the computing system 1400 to determine whether one or more images captured by one or more cameras indicates that a gesture has been formed.

[0081] The computing system 1400 can include the gesture recognizer 1404. The gesture recognizer 1404 can recognize gestures, such as gestures formed by the hand 106. The gesture recognizer 1404 can recognize the gestures based on one or multiple images captured by the camera 110 and / or processed by the environmental monitor 1402. The gesture recognizer 1404 can recognize the gestures byimplementing a three-dimensional model-based algorithm, by implementing a skeletal-based algorithm, or by implementing an appearance-based model, as nonlimiting examples. In some implementations, the gesture recognizer 1404 recognizes gestures based on relative positions of the thumb 212 and forefinger 214 of the hand 106 of the user 102 and / or an orientation of the thumb 212 with respect to the forefinger 214. In some implementations, the gesture recognizer 1404 can recognize the open gesture formed by the hand 106. In some implementations the gesture recognizer 1404 recognizes the open gesture based on an angle between the thumb 212 and the forefinger 214 satisfying an open gesture angle threshold. In some implementations, the gesture recognizer 1404 can recognize the pinching gesture formed by the hand 106. In some implementations, the gesture recognizer 1404 can recognize the pinching gesture based on an end portion of the thumb 212 contacting an end portion of the forefinger 214.

[0082] The gesture recognizer 1404 can include a duration determiner 1406. The duration determiner 1406 can determine a duration that a gesture is maintained and / or held. The gesture for which the duration determiner 1406 determines the duration of maintenance and / or holding can be an open gesture or a pinching gesture, as non-limiting examples. The duration determiner 1406 can measure a time duration from which a gesture began and / or was initiated until a present time. The duration determiner 1406 can measure, for example, how long an open gesture or a pinching gesture has been maintained or held. The duration determiner 1406 can measure a time duration from which a gesture ceased to be presented until a present time. The duration determiner 1406 can measure, for example, a time duration since a hand stopped resenting an open gesture or released a pinching gesture.

[0083] In some implementations, the gesture recognizer 1404 determines whether to recognize a gesture based in part on a time duration determined by the duration determiner 1406. For example, the gesture recognizer 1404 can determine to recognize a pinching gesture, and / or a graphical user interface (GUI) generator 1410 can determine to generate and / or present a graphical user interface such as the graphical user interface 208, graphical user interface 1208, or graphical user interface 1308, based on the gesture recognizer 1404 determining that a head-mounted device 104 held a pinching gesture for a duration of time that satisfies a hold time condition (such as holding the pinching gesture for at least a hold time threshold duration). Insome implementations, the gesture recognizer 1404 and / or GUI generator 1410 causes the GUI to be maintained and / or presented while the pinching gesture is maintained and / or based on a duration of time that the GUI generator 1410 determines that the pinching gesture is maintained. In some implementations, the duration determiner 1406 can determine whether the duration of the pinching gesture satisfies an information time condition. If the duration determiner 1406 determines that the duration of the pinching satisfies the information time condition, such as based on the duration of time of the pinching gesture meeting or exceeding an information time threshold, then the duration determiner 1406 and / or GUI generator 1410 can cause the GUI to include information about an action associated with an icon included in the GUI.

[0084] The gesture recognizer 1404 can include a movement determiner 1408. The movement determiner 1408 can determine whether a hand, such as the hand 106 of the user 102, is moving. The movement determiner 1408 can determine whether the hand 106 is moving based on comparing images of the hand 106 captured by a camera such as the camera 110. The movement determiner 1408 can determine whether the hand 106 is moving by, for example, comparing differences between images of the hand 106 to a movement threshold. In some implementations, the movement determiner 1408 and / or GUI generator 1410 can wait to present and / or generate the GUI until after the hand 106 is stationary and / or not moving. In some implementations, the movement determiner 1408 and / or GUI generator 1410 will not present and / or generate the GUI if the hand 106 is moving. In some implementations, the movement determiner 1408 and / or GUI generator 1410 can present and / or generate the GUI based on determining that the hand 106 is stationary and / or not moving as well as forming the pinching gesture.

[0085] The movement determiner 1408 can determine movements of a hand 106, and / or portion of the hand 106 forming the pinching gesture (such as portions of the thumb 212 and forefinger 214 in contact with each other). In some implementations, the movement determiner 1408 determines movements of the hand 106 and / or portion of the hand 106 with respect to a GUI such as the graphical user interface 208, graphical user interface 1208, and / or graphical user interface 1308. The movement determiner 1408 can determine a direction of movement of the hand 106, such as an icon toward which the hand 106 and / or portion of the hand 106 moved.

[0086] The computing system 1400 can include the GUI generator 1410. The GUI generator 1410 can generate and / or present a GUL such as the graphical user interface 108, graphical user interface 208, graphical user interface 1208, and / or graphical user interface 1308. The GUI generator 1410 can generate and / or present a portion of a GUI, such as the first portion 218, second portion 228, or portion 1218.

[0087] The GUI generator 1410 can generate and / or present the GUI based on gestures, such as gestures recognized by the gesture recognizer 1404. The GUI generator 1410 can, for example, generate a GUI based on the gesture recognizer recognizing a pinching gesture. In some implementations, the GUI generator 1410 can generate the GUI based on the gesture recognizing the pinching gesture and the duration determiner 1406 determining that a time of the pinching gesture being held satisfies a hold time condition. The hold time condition can be a duration of the pinching gesture being held being at least a hold threshold time. In some implementations, the GUI generator 1410 ceases presenting the GUI based on the gesture recognizer 1404 recognizing that the hand 106 releases the pinching gesture.

[0088] In some implementations, the GUI generator 1410 generates and / or presents a portion of the GUI based on the gesture recognizer 1404 recognizing the hand 106 forming the open gesture. The GUI generator 1410 can present the portion of the GUI based on the gesture recognizer 1404 recognizing the hand 106 forming the open gesture and present remaining portions of the GUI based on the gesture recognizer 1404 recognizing the hand 106 forming the pinching gesture. In some implementations, the GUI generator 1410 ceases presenting portions of the GUI other than a selected icon for a predetermined time duration. After the predetermined time duration the GUI generator 1410 ceases presenting the selected icon or any portion of the GUI.

[0089] In some implementations, the GUI generator 1410 maintains a position of the GUI with respect to the hand 106, and / or with respect to the thumb 212 and forefinger 214. while the hand 106 and / or thumb 212 and forefinger 214 move with respect to one or more background objects. In some implementations, the GUI generator 1410 maintains an orientation of the GUI with respect to one or more background objects while the camera 110 moves and / or rotates with respect to the one or more background objects.

[0090] The computing system 1400 can include an action selector 1412. Theaction selector 1412 can select an action based on gestures of the hand 106. The action selector 1412 can, for example, select an action based on a selected icon. The selected icon can be an icon over which the hand 106 is held when a pinching gesture is released and / or an icon toward which the hand 106 moved before the pinching gesture was released. The action selector 1412 can select an action associated with the icon over which the hand 106 was held and / or toward which the hand 106 was moved. The action selector 1412 can select an action based on a direction of recognized movement, such as movement recognized by the movement determiner 1408, to or over an icon.

[0091] The computing system 1400 can include an action processor 1414. The action processor 1414 can process and / or perform actions. In some implementations, the actions processed and / or performed by the action processor 1414 are actions associated with the selected icon. In some implementations, the actions processed and / or performed by the action processor 1414 are actions selected by the action selector 1412. In some implementations, the actions include presenting a home screen, presenting a previous screen or set of icons, or presenting an overview. In some implementations, the actions are actions associated with a selected icon within an application that is executing on the computing system 1400 or another computing system in communication with the computing system 1400.

[0092] The computing system 1400 can include at least one processor 1416. The at least one processor 1416 can execute instructions, such as instructions stored in at least one memory device 1418, to cause the computing system 1400 to perform any combination of methods, functions, and / or techniques described herein.

[0093] The computing system 1400 can include at least one memory device 1418. The at least one memory device 1418 can include anon-transitory computer- readable storage medium. The at least one memory device 1418 can store data and instructions thereon that, when executed by at least one processor, such as the processor 1416, are configured to cause the computing system 1400 to perform any combination of methods, functions, and / or techniques described herein. Accordingly, in any of the implementations described herein (even if not explicitly noted in connection with a particular implementation), software (e g., processing modules, stored instructions) and / or hardware (e.g., processor, memory devices, etc.) associated with, or included in, the computing system 1400 can be configured to perform, alone.or in combination with the computing system 1400, any combination of methods, functions, and / or techniques described herein. The at least one memory device 1418 can include a gesture library. The gesture 11 bran can include predetermined gestures. The predetermined gestures can include hand formations and / or movements and associated actions.

[0094] The computing system 1400 may include at least one input / output node 1420. The at least one input / output node 1420 may receive and / or send data, such as from and / or to, a server, and / or may receive input and provide output from and to a user. The input and output functions may be combined into a single node, or may be divided into separate input and output nodes. The input / output node 1420 can include a microphone, a camera (such as a front-facing camera), an IMU, a display, a speaker, one or more buttons, and / or one or more wired or wireless interfaces for communicating with other computing devices such as the head-mounted device 104 and / or a computing device in communication with the head-mounted device 104.

[0095] FIGs. 15 A, 15B, and 15C show an implementation of the headmounted device 104. As shown in FIGs. 15A, 15B, and 15C, the head-mounted device 104 includes a frame 1502. The frame 1502 includes a front frame portion defined by rim portions 1503 A, 1503B surrounding respective optical portions in the form of lenses 1507 A, 1507B, with a bridge portion 1509 connecting the rim portions 1503A, 1503B. Arm portions 1505 A, 1505B are coupled pivotably or rotatably coupled, to the front frame by hinge portions 1512A, 1512B at the respective rim portion 1503 A, 1503B. In some implementations, the lenses 1507A, 1507B may be corrective / prescription lenses. In some implementations, the lenses 1507 A, 1507B may be an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters. Displays 1510A, 1510B may be coupled in a portion of the frame 1502. In the implementation shown in FIG. 15B, the displays 1510A, 1510B are coupled in the arm portions 1505A, 1505B and / or rim portions 1503A, 1503B of the frame 1502. In some implementations, the head-mounted device 104 can also include an audio output device 1516 (such as one or more speakers), an illumination device 1518, at least one processor 1511 (which is an example of the processor 1416 show n and described with respect to FIG. 14), an outward-facing image sensor 1514 (or camera 110), and gaze-tracking cameras 1519A, 1519B that can capture images of eyes of the user 102 to track a gaze of theuser 102. In some implementations, the head-mounted device 104 may include a see- through near-eye display. The displays 1510A, 1510B may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world through the lenses 1507A, 1507B, next to content (such as digital images, user interface elements, virtual content, and the like) generated by the displays 1 10A, 1510B. In some implementations, waveguide optics may be used to depict content on the displays 1510A, 1510B via outcoupled light 1520A, 1520B. The images projected by the displays 1510A, 1510B onto the lenses 1507A, 1507B may be translucent, allowing the user 102 to see the images projected by the displays 1510A, 151 OB as well as physical objects beyond the head-mounted device 104.

[0096] FIG. 15D shows another implementation of the head-mounted device 104. In this implementation, the head-mounted device 104 is in goggle form, with a display included in the head-mounted device 104 and a housing supporting the display enclosing the face and / or eyes of the user 102. This implementation of the headmounted device 104 can support a virtual reality (VR) experience in which the user 102 sees only what is presented by the display included in the head-mounted device 104.

[0097] FIG. 16 is a flowchart showing a method performed by a computing system. The computing system can include the computing system 1400 and / or the head-mounted device 104. The method includes recognizing a pinching gesture (1602). Recognizing a pinching gesture (1602) can include recognizing the pinching gesture formed by a thumb and forefinger. The method includes presenting a graphical user interface (1604). Presenting the graphical user interface (1604) can include based on recognizing the pinching gesture, presenting the graphical user interface. The method includes recognizing movement of the thumb and forefinger (1606). Recognizing movement of the thumb and forefinger (1606) can include, while presenting the graphical user interface, recognizing movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture. The method includes selecting an action (1608). Selecting the action(1608) can include, based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, selecting the action. The action can be selected based on a direction of the recognized movement. The method includes performing the action (1610).

[0098] In some implementations, presenting the graphical user interface is based on recognizing the pinching gesture being held for a time that satisfies a hold time condition.

[0099] In some implementations, the method further includes recognizing an open gesture based on an orientation of the thumb with respect to the forefinger; based on recognizing the open gesture, presenting a portion of the graphical user interface, and wherein presenting the graphical user interface is presenting an additional portion of the graphical user interface after presenting the portion of the graphical user interface.

[0100] In some implementations, the open gesture is recognized based on the orientation of the thumb and the forefinger and independently of orientations of fingers other than the thumb and forefinger on a hand that includes the thumb and forefinger.

[0101] In some implementations, the method further includes determining that the open gesture was maintained for an open delay time. Recognizing the open gesture is based on recognizing the open gesture based on the orientation of the thumb with respect to the forefinger and determining that the open gesture was maintained for the open delay time.

[0102] In some implementations, recognizing the open gesture is further based on determining that a palm of the hand is facing a camera that captured an image of the hand.

[0103] In some implementations, the method further includes recognizing a release of the pinching gesture and ceasing presenting the graphical user interface based on recognizing the release of the pinching gesture.

[0104] In some implementations, ceasing presenting the graphical user interface includes ceasing presenting portions of the graphical user interface other than a selected icon, the selected icon being associated with the action; and after ceasing presenting portions of the graphical user interface other than the selected icon, ceasing presenting the selected icon.

[0105] In some implementations, a duration of time that the graphical user interface is maintained is based on a duration of time that the pinching gesture is maintained.

[0106] In some implementations, the method further includes maintaining a position of the graphical user interface with respect to the thumb and forefinger while the thumb and forefinger move with respect to a background object.

[0107] In some implementations, the method further includes maintaining an orientation of the graphical user interface with respect to the background object while an orientation of a head of a user changes with respect to the background object, the head-mounted device being mounted on the head of the user.

[0108] In some implementations, the method further includes determining that a duration of time that a pinching gesture has been maintained satisfies a pinching time condition; and based on the satisfaction of the pinching time condition, presenting information associated with an icon included in the graphical user interface.

[0109] In some implementations, the method further includes determining that a hand is stationary the hand including the thumb and the forefinger, wherein presenting the graphical user interface is based on determining that the hand is stationary’ and recognizing the pinching gesture.

[0110] Implementations of the various techniques described herein may be implemented in digital electronic circuitry; or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0111] Method steps may be performed by one or more programmableprocessors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0112] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory’, including by way of example semiconductor memory devices, e.g.. EPROM. EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.

[0113] To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory' feedback, e.g., visual feedback, auditory7feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end.middleware, or front-end components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0115] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the embodiments of the invention.

Claims

WHAT IS CLAIMED IS:

1. A method performed by a head-mounted device, the method comprising: recognizing a pinching gesture formed by a thumb and forefinger; based on recognizing the pinching gesture, presenting a graphical user interface; while presenting the graphical user interface, recognizing movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, selecting an action, the action being selected based on a direction of the recognized movement; and performing the action.

2. The method of claim 1, wherein presenting the graphical user interface is based on recognizing the pinching gesture being held for a time that satisfies a hold time condition.

3. The method of either of claims 1 or 2, further comprising: recognizing an open gesture based on an orientation of the thumb with respect to the forefinger; based on recognizing the open gesture, presenting a portion of the graphical user interface, and wherein presenting the graphical user interface is presenting an additional portion of the graphical user interface after presenting the portion of the graphical user interface.

4. The method of claim 3, wherein the open gesture is recognized based on the orientation of the thumb and the forefinger and independently of orientations of fingers other than the thumb and forefinger on a hand that includes the thumb and forefinger.

5. The method of either of claims 3 or 4, further comprising: determining that the open gesture was maintained for an open delay time, wherein recognizing the open gesture is based on recognizing the open gesture based on the orientation of the thumb with respect to the forefinger and determining that the open gesture was maintained for the open delay time.

6. The method of any of claims 3-5, wherein recognizing the open gesture is further based on determining that a palm of the hand is facing a camera that captured an image of the hand.

7. The method of any of the preceding claims, further comprising: recognizing a release of the pinching gesture; and ceasing presenting the graphical user interface based on recognizing the release of the pinching gesture.

8. The method of claim 7, wherein ceasing presenting the graphical user interface includes: ceasing presenting portions of the graphical user interface other than a selected icon, the selected icon being associated with the action; and after ceasing presenting portions of the graphical user interface other than the selected icon, ceasing presenting the selected icon.

9. The method of any of the preceding claims, wherein a duration of time that the graphical user interface is maintained is based on a duration of time that the pinching gesture is maintained.

10. The method of any of the preceding claims, further comprising maintaining a position of the graphical user interface with respect to the thumb and forefinger while the thumb and forefinger move with respect to a background object.

11. The method of any of the preceding claims, further comprising maintaining an orientation of the graphical user interface with respect to thebackground object while an orientation of a head of a user changes with respect to the background object, the head-mounted device being mounted on the head of the user.

12. The method of any of the preceding claims, further comprising: determining that a duration of time that a pinching gesture has been maintained satisfies a pinching time condition; and based on the satisfaction of the pinching time condition, presenting information associated with an icon included in the graphical user interface.

13. The method of any of the preceding claims, further comprising: determining that a hand is stationary, the hand including the thumb and the forefinger, wherein presenting the graphical user interface is based on determining that the hand is stationary and recognizing the pinching gesture.

14. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to perform the method of any of the preceding claims.

15. A computing system comprising: at least one processor; and a non-transitory’ computer-readable storage medium comprising instructions stored thereon that, when executed by the at least one processor, are configured to cause the computing system to perform the method of any of claims 1-13.

16. A head-mounted device comprising: an image sensor configured to capture images of a hand; a display configured to present a graphical user interface; at least one processor; and a non-transitory' computer-readable storage medium comprising instructions stored thereon that, when executed by the at least one processor, are configured to cause the head-mounted device to:recognize a pinching gesture formed by a thumb and forefinger based on a first set of images captured by the image sensor; based on recognizing the pinching gesture, present, via the display, a graphical user interface; while presenting the graphical user interface, recognize movement of the thumb and forefinger while the thumb and forefinger maintain and subsequently release the pinching gesture based on a second set of images captured by the image sensor; based on recognizing movement and subsequent release of the pinching gesture by the thumb and forefinger, select an action, the action being selected based on a direction of the recognized movement; and perform the action.

Images