Method and apparatus for applying free space input for surface constrained control

Abstract

To provide a method and apparatus for applying free space input for surface constrained control.SOLUTION: A free space input standard is instantiated on a processor. Free space input is sensed and communicated to a processor. If the free space input satisfies the free space input criteria, a touch screen input response is invoked to the operating system. Free space input may be sensed using a continuous implicit, discrete implicit, active explicit, or passive explicit approach. The touch screen input response may be invoked through the transmission of a virtual touch screen input, virtual input event, or virtual command to or within the operating system. In this manner, free space gestures may control existing touch screen interfaces and devices to accept free space gestures without direct modification of such interfaces and devices.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent No. 14 / 713,971, filed May 15, 2015, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to controlling electronic devices using poses and / or gestures. In particular, the present disclosure relates to using free-space (e.g., substantially unrestricted and / or three-dimensional) poses, gestures, and / or other inputs to control devices configured to invoke and respond to surface-only input functionality. [Background technology]

[0003] Input can be provided by surface-limited manipulation, for example, by use of so-called "touch screen" interfaces. Touch screens incorporate a touch-sensitive surface that overlays a visual display. Such configurations allow a user to provide input by physical contact, for example, by touching the touch screen with the tip of a finger and moving the finger while keeping it in contact with the touch screen.

[0004] Various individual contacts or sets of contacts with a touchscreen can be assigned to correspond to particular functions of a device or system communicating with the touchscreen. For example, touching the screen with the tip of a finger, sliding the fingertip to the right while keeping it in contact with the touchscreen, and lifting the fingertip off the surface of the touchscreen can be collectively referred to as a "right swipe" and associated with a function such as "move the screen content to the right."

[0005] However, physical contact input and / or surface-limited input is not desirable in all situations. For example, at least in certain cases, there may be concerns about the potential for contamination or damage to the touchscreen, the difficulty of holding the physical screen while providing input, etc. Also, touchscreens are typically limited to receiving only input that can occur in a finite, two-dimensional surface. Summary of the Invention

[0006] Embodiments relate to various systems, devices, methods, and paradigms for applying free-space input for surface-limited control.

[0007] In one embodiment, a method is provided that includes setting a free-space input criterion in a processor, sensing a free-space input with a sensor, and notifying the processor of the free-space input, wherein if the free-space input satisfies the free-space input criterion, a surface-limited input response of the surface-limited input is invoked in the processor.

[0008] The free space input criteria can be three-dimensional input criteria. The free space input criteria can be gesture criteria and / or pose criteria. The free space input criteria can include a motion criterion of the end effector in free space.

[0009] Surface-only input can include touchscreen input, touchdown, and / or touchdown and swipe.

[0010] Sensing the free-space input can include imaging, stereo imaging, video imaging, depth imaging the free-space input, and can include evaluating over a period of time, and can include evaluating multiple images at time intervals over a period of time. The device can be controlled using surface-limited input responses.

[0011] The free-space input criteria can include multiple end-effector criteria for at least one end-effector considered together for the end-effector with respect to the free-space input. The free-space input criteria can include, for example, various criteria related to speed, velocity, position, orientation, direction, acceleration, joint angles, separation, extension, and / or selection of the end-effector.

[0012] The free-space input criteria can include a profile of a lateral extended finger swipe that takes into account path shape, velocity, acceleration, and joint position over a period of time. The free-space input criteria can include various criteria, such as speed, velocity, position, orientation, direction, acceleration, joint angle, end-effector separation, extension, and / or selection criteria.

[0013] The free-space input criteria may include a plurality of end-effector motion criteria of the end-effector that are considered consecutively with respect to at least first and second sub-criteria, the first sub-criterion corresponding to a first portion of the surface-limited input and the second sub-criterion corresponding to a second portion of the surface-limited input, the first and second portions being considered consecutively with respect to the free-space input.

[0014] The free space input criteria can include criteria for the effector contacting a bounded region, where the end effector contacting the bounded region corresponds to contacting the bounded surface of the surface-bounded input.

[0015] The bounding region can include a spherical hull, a flat surface, and / or a rectilinear solid. The bounding region can be at least approximately aligned with a surface of a physical object. The free-space input criteria can include criteria related to the speed, velocity, position, orientation, direction, acceleration, joint angle, separation, extension, and / or selection of the end effector.

[0016] The free-space input criteria can include a plurality of end-effector movement criteria related to the end-effector and an indication that is substantially simultaneous with the movement of the end-effector. The indication can include hand pose, hand gesture, eye pose, eye gesture, and / or voice command. The free-space input criteria can include criteria related to the speed, velocity, position, orientation, direction, acceleration, joint angle, separation, extension, and / or selection of the end-effector.

[0017] The free-space input criterion can include multiple state criterion. The state criterion can be configured to be combined to form multiple free-space input criterion, each free-space input criterion corresponding to one free-space input. Each state criterion can include a motion criterion for the end effector.

[0018] In another embodiment, an apparatus is provided that includes means for setting a free-space input criterion in a processor, means for sensing the free-space input with a sensor, and means for notifying the processor of the free-space input, the apparatus further including means for invoking a surface-limited input response to the surface-limited input in the processor if the free-space input satisfies the free-space input criterion.

[0019] In another embodiment, an apparatus is provided that includes a processor configured to execute executable instructions and a sensor in communication with the processor, the apparatus including a free-space input criterion instantiated in the processor, and free-space input comparison means instantiated in the processor and configured to compare the free-space input with the free-space input criterion, the apparatus further including surface-limited input response invoking means instantiated in the processor and configured to invoke a surface-limited input response to the surface-limited input on the processor.

[0020] The processor and the sensor may be located in the head mounted display.

[0021] In another embodiment, a method is provided that includes setting a free-space input criterion in a processor, sensing a free-space input with a sensor, and notifying the processor of the free-space input, the method includes generating a surface-limited communication in the processor, and notifying a data entity of the surface-limited communication, wherein if the free-space input satisfies the free-space input criterion, the data entity performs a reaction corresponding to the surface-limited input.

[0022] The data entity may include an operating system and / or may be instantiated on a processor.

[0023] The free space input may include three-dimensional input and / or may include hand poses and / or hand gestures.

[0024] The method may include generating a virtual surface-limited input in a processor and notifying a data entity of the virtual surface-limited input, wherein if the free space input satisfies a free space input criterion, the data entity accepts the virtual surface-limited input as a surface-limited input and performs a reaction corresponding to the surface-limited input. Notifying the data entity of the virtual surface-limited input may include establishing a virtual surface-limited input link and identifying the virtual surface-limited input link to the data entity as the actual surface-limited input source.

[0025] The virtual surface-limited input can include a virtual touchscreen input. The virtual surface-limited input can include a virtual touchdown input. The virtual surface-limited input can include a virtual touchdown and swipe input. The virtual surface-limited input can include at least x and y coordinates and pressure over a period of time that generally correspond to touchdowns and swipes on a touchscreen.

[0026] The method can include generating a virtual surface-limited input taking into account the free space input. The virtual surface-limited input can include converting the free space input into a corresponding surface-limited form. The method can include generating the virtual surface-limited input excluding the free space input.

[0027] The method may include having a processor generate a virtual surface-bound input event for the data entity and notifying the data entity of the virtual surface-bound input event, wherein if the free space input satisfies a free space input criterion, the data entity accepts the virtual surface-bound input event as a surface-bound input event and performs a reaction corresponding to the surface-bound input event. Notifying the data entity of the virtual surface-bound input event may include establishing a virtual event link and identifying the virtual event link to the data entity as the actual event source.

[0028] The virtual surface limited input events can include virtual touchscreen input events, virtual touchdown input events, and / or virtual touchdown and swipe input events.

[0029] The method may include causing a processor to generate a virtual command for the data entity and communicating the virtual command to the data entity, wherein if the free-space input satisfies a free-space input criterion, the data entity accepts the virtual command as a command and performs a reaction corresponding to the command. Communicating the virtual command to the data entity may include establishing a virtual command link and identifying the virtual command link to the data entity as the actual command source.

[0030] The virtual commands may include virtual touchscreen commands, virtual touchdown commands, and / or virtual touchdown and swipe commands.

[0031] The method can include using the reaction to control a device.

[0032] In another embodiment, a method is provided that includes establishing free-space input criteria in a processor of a head-mounted display, including criteria related to speed, velocity, direction, acceleration, joint angle, finger separation, finger extension, and / or finger selection of a hand gesture in free space. The method includes sensing a plurality of depth images of the free-space input of the hand gesture over a period of time using a depth imaging device of the head-mounted display, and reporting the free-space input to the processor. If the free-space input satisfies the free-space input criteria, virtual touchscreen touchdown and swipe input is generated in the processor, and the virtual touchscreen touchdown and swipe input is reported to an operating system, which executes a response thereto and controls the head-mounted display accordingly.

[0033] In another embodiment, an apparatus is provided that includes means for setting a free-space input criterion on a processor, means for sensing the free-space input, and means for notifying the processor of the free-space input, the apparatus further including, if the free-space input satisfies the free-space input criterion, generating a virtual surface-limited communication to the processor, notifying a data entity of the surface-limited communication, and causing the data entity to perform a reaction corresponding to the communication.

[0034] In another embodiment, an apparatus is provided that includes a processor and a sensor in communication with the processor, the apparatus including a free-space input criterion instantiated on the processor, and a free-space input comparison means instantiated on the processor and configured to compare the free-space input with the free-space input criterion, the apparatus further including a virtual surface-limited communication generation means instantiated on the processor and configured to generate a surface-limited communication, and a communication means instantiated on the processor and configured to communicate the surface-limited communication to a data entity, whereby the data entity performs a reaction corresponding to the surface-limited input.

[0035] The processor and the sensor may be located in a head mounted display. The sensor may include a depth imager. [Brief explanation of the drawings]

[0036] Like reference numerals generally refer to corresponding elements throughout the drawings.

[0037] [Figure 1A] FIG. 1A is a top view illustrating an example of a touchscreen input in the form of a right swipe. [Figure 1B] FIG. 1B is a perspective view illustrating an example of a touchscreen input in the form of a right swipe.

[0038] [Figure 2A] FIG. 2A is a perspective view showing individual events in a touchscreen input. [Figure 2B] FIG. 2B is a perspective view showing individual events in a touchscreen input. [Figure 2C] FIG. 2C is a perspective view showing individual events in a touchscreen input. [Figure 2D] FIG. 2D is a perspective view showing individual events in a touchscreen input.

[0039] [Figure 3] FIG. 3 is a top view showing an example configuration for free space input.

[0040] [Figure 4A] FIG. 4A is a front perspective view showing another exemplary configuration for free space input. [Figure 4B] FIG. 4B is a rear perspective view showing another exemplary configuration for free space input.

[0041] [Figure 5] FIG. 5 is a perspective view showing an example initial hand position for free space input.

[0042] [Figure 6] FIG. 6 illustrates an exemplary configuration of free-space input that invokes surface-limited input in a continuous implicit format.

[0043] [Figure 7] FIG. 7 shows an example configuration for a free space input that illustrates one example of an exclusion phenomenon that can be considered in the criteria.

[0044] [Figure 8] FIG. 8 shows another exemplary configuration for a free-space input illustrating another example of an exclusion phenomenon that can be considered in the criteria.

[0045] [Figure 9] FIG. 9 illustrates an example configuration of free-space input that invokes surface-only input in a discrete implicit format.

[0046] [Figure 10] FIG. 10 illustrates the construction of free-space input that invokes surface-only input in a passive-explicit format and incorporates bounding surfaces.

[0047] [Figure 11A] FIG. 11A illustrates the construction of free-space input that invokes surface-only input in a passive-explicit format and incorporates bounding volumes. [Figure 11B] FIG. 11B illustrates the construction of free-space input that invokes surface-only input in a passive-explicit format and incorporates bounding volumes. [Figure 11C] FIG. 11C illustrates the construction of free-space input that invokes surface-only input in a passive-explicit format and incorporates bounding volumes. [Figure 11D] FIG. 11D illustrates the construction of free-space input that invokes surface-only input in a passive-explicit format and incorporates bounding volumes.

[0048] [Figure 12A] FIG. 12A illustrates the construction of free-space input that invokes surface-limited input in an active-explicit format. [Figure 12B] FIG. 12B illustrates the construction of free-space input that invokes surface-limited input in an active-explicit format. [Figure 12C] FIG. 12C illustrates the construction of free-space input that invokes surface-limited input in an active-explicit format. [Figure 12D] FIG. 12D illustrates the construction of free-space input that invokes surface-limited input in an active-explicit format.

[0049] [Figure 13] FIG. 13 is a flowchart illustrating an exemplary method for applying free-space input to surface-limited control.

[0050] [Figure 14] FIG. 14 is a flow chart illustrating another exemplary method for applying free-space input to surface-limited control using a continuous implicit approach.

[0051] [Figure 15A] FIG. 15A is a flowchart illustrating another exemplary method for applying free-space input to surface-limited control using a discrete implicit approach. [Figure 15B] FIG. 15B is a flowchart illustrating another exemplary method for applying free-space input to surface-limited control using a discrete implicit approach.

[0052] [Figure 16] FIG. 16 is a perspective view illustrating an example configuration for sensing free-space input using a continuous implicit approach.

[0053] [Figure 17] FIG. 17 is a perspective view illustrating another exemplary configuration for sensing free-space input using a discrete implicit approach.

[0054] [Figure 18] FIG. 18 is a flow chart illustrating another exemplary method for applying free-space input to surface-limited control using a passive-explicit approach.

[0055] [Figure 19]FIG. 19 is a flow chart illustrating another exemplary method for applying free-space input to surface-limited control using an active-explicit approach.

[0056] [Figure 20] FIG. 20 is a flow chart illustrating an exemplary method for applying free-space input to surface-limited control and extending response through virtual input.

[0057] [Figure 21] FIG. 21 is a flowchart illustrating an exemplary method for applying free-space input to surface-limited control and evoking a response through virtual input to a head-mounted display.

[0058] [Figure 22] FIG. 22 is a flowchart illustrating an exemplary method for applying free-space input to surface-limited control and evoking a response through virtual events to a head-mounted display.

[0059] [Figure 23] FIG. 23 is a flowchart illustrating an exemplary method for applying free-space input to surface-limited control and invoking a response through virtual commands to a head-mounted display.

[0060] [Figure 24A] FIG. 24A is a flowchart illustrating an exemplary method for applying free-space input to surface-bound control using a discrete implicit approach and invoking responses through virtual commands to a head-mounted display. [Figure 24B] FIG. 24B is a flowchart illustrating an exemplary method for applying free-space input to surface-bound control using a discrete implicit approach and invoking responses through virtual commands to a head-mounted display.

[0061] [Figure 25] FIG. 25 is a schematic diagram illustrating an exemplary embodiment of the device.

[0062] [Figure 26] FIG. 26 is a schematic diagram illustrating another exemplary embodiment of the device with additional components.

[0063] [Figure 27] FIG. 27 is a schematic diagram illustrating an exemplary embodiment of the device.

[0064] [Figure 28] FIG. 28 is a block diagram of a processing system in which the operations may be performed. DETAILED DESCRIPTION OF THE INVENTION

[0065] Various embodiments of the present disclosure address using free-space input to invoke surface-only input, e.g., using three-dimensional gestures in space to somehow perform commands mapped to touchscreen input. Thus, it may be key to consider the specific characteristics of surface-only input, for example, using a touchscreen "swipe right" input. A right swipe applied as input to certain touchscreens and / or touchscreen-controlled devices (e.g., smartphones, tablet computers), may, for example, move displayed content to the right, "flip the page" in an e-book, or perform other functions.

[0066] 1A is a top view of input provided through a touchscreen in the form of a right swipe. As shown, for a right swipe, hand 104A moves from left to right as indicated by motion arrow 106A (shown for clarity, but not necessarily actually visible) while hand 104A is in contact with touchscreen 102A. As shown in the configuration of FIG. 1A, touchscreen 102A is part of a tablet computer, but this is merely an example.

[0067] 1B next shows a similar right swipe, this time from a perspective view slightly above and in front of touchscreen 102B. Again, as shown, hand 104B is in contact with touchscreen 102B (more specifically, the tip of an extended index finger is in contact with touchscreen 102B) and is moving from left to right as indicated by motion arrow 106B (left to right from the viewer's perspective, right to left in the perspective view).

[0068] When considered purely as a motion, a right swipe appears simple in nature, being a left-to-right motion. However, in reality, input provided via a touchscreen interface is not necessarily a simple motion. Typically, operating systems and other programs or executable instructions supporting touchscreens address touchscreen input as several distinct events. FIGS. 2A-2D show an example breakdown of specific events that may exist for a right swipe. However, it should be understood that the events shown in FIGS. 2A-2D are not necessarily exhaustive, and additional and / or different events may be considered for a particular touchscreen interface. Furthermore, a right swipe is only one of many possible inputs, and other inputs may be considered other events.

[0069] Next, Figure 2A is another perspective view of touchscreen 202A. The configuration in Figure 2A represents an initial state before a right swipe is input to the touchscreen, with hand 204A hovering above touchscreen 202A and not in contact with touchscreen 202A at the time of illustration.

[0070] Continuing to refer to Figure 2B, as can be seen by comparison with Figure 2A, hand 204B has moved down toward and contacted touchscreen 202B. The downward movement is also indicated by movement arrow 206B. Figure 2B also shows contact marker 208B, which represents the point of contact between hand 204B and touchscreen 202B. At this point, hand 204B has not yet made a left-to-right movement across the face of touchscreen 202B.

[0071] Contact marker 208B is shown to emphasize that actual contact exists between hand 204B and touchscreen 202B (contact marker 208B is an example and may not actually be visible). Contact is typically very important in touchscreen input, not just as a way to track movement across the surface of touchscreen 202B, but as an input event in its own right. This importance is addressed in more detail with reference to the remainder of FIG. 2.

[0072] 2C, as can be seen by comparison with FIG. 2B, hand 204C moves across the surface of touchscreen 202C from left to right (from the viewer's perspective) while in contact with the surface. The left-to-right movement is also indicated by movement arrow 206C. As shown and highlighted by contact marker 208C, hand 204C remains in contact with touchscreen 202C and maintains contact with touchscreen 202C during the movement indicated by movement arrow 206C.

[0073] Continuing to refer to Figure 2D, as can be seen by comparison with Figure 2C, hand 204D has moved upward off the surface of touchscreen 202D and is no longer in contact with touchscreen 202D. The upward movement is also indicated by movement arrow 206D. Contact marker 208D is shown for illustrative purposes to indicate the last point of contact between hand 204D and touchscreen 202D, but said contact is no longer ongoing.

[0074] While the phrase "right swipe" specifies a rightward motion, in reality, a right swipe may consist of multiple events. For example, as collectively shown in Figures 2A-2D, a right swipe can be conceptualized as incorporating, for example, a touch-down event representing hand 204B contacting touchscreen 202B at contact point 208B as shown in Figure 2B, a swipe event representing a swiping motion in which hand 204C moves across touchscreen 202C while maintaining contact with touchscreen 202C at contact marker 208C as shown in Figure 2C, and a lift-off event in which hand 204D leaves contact with touchscreen 202D from previous contact point 208D.

[0075] 2A-2D are merely examples, touchscreens may actually utilize similar multiple event configurations when receiving input. That is, a touchscreen may require both a touch-down event and a swipe event (and / or a lift-off event, etc.). Because these events may be sensed individually and processed separately by the processor (and / or operating system or other programs instantiated on the operating system), a swipe event alone may not be recognized as a swipe input if the touch-down event, lift-off event, or both are not present (or are not somehow properly associated with the swipe).

[0076] Thus, even if a swipe action itself is considered to "define" a swipe from a conversational perspective, from an execution perspective, a swipe input may require one or more additional events, such as a touchdown and / or liftoff event. Typically, this may be a practical consideration, and it may be useful to require a clear start (touchdown) and / or a clear end (liftoff) of a swipe to reliably distinguish a swipe from other inputs, noise, etc.

[0077] As noted above, various embodiments address using free-space input to invoke surface-limited input. Continuing with the example of Figures 2A-2D, using a three-dimensional hand swipe gesture can include causing a processor, processor-controlled device, system, etc., to perform a function associated with a swipe input by the processor to a touchscreen.

[0078] At least certain advantages of the above configuration relate to the ability to apply existing hardware, software, know-how, etc. relating to touchscreens and other surface-limited input interfaces to new devices, interfaces, input and control methods, etc.

[0079] For example, consider an electronic device, such as a smartphone, that is already configured to accept input from a touchscreen. Such a device may operate, at least in part, through an operating system, such as the Android OS, instantiated on a processor. In the case of Android as a specific example, support already exists for recognizing touchscreen inputs, such as swipes, sensing the events that constitute a gesture (e.g., touchdown, swipe, etc., in the case of the right swipe described above), and executing various commands and / or other functions within the device running Android in response to the touchscreen inputs.

[0080] Similarly, given the existence of an electronic device configured to accept input from a touchscreen, additional programs, plug-ins, data files, and other entities other than the operating system may also be available with the device, again taking Android as an example, including media players, games, messaging applications, video conferencing programs, and the like.

[0081] Typically, operating systems and / or other programs configured for use with a device or system that accepts touchscreen input may include functionality to recognize, receive, and react to such touchscreen input. However, such operating systems and / or other programs typically cannot be configured to accept input, such as gestures in free space, using new approaches.

[0082] Existing smartphones routinely cannot respond to hand gestures. This is true even if the smartphone in principle had an appropriate sensor, such as a camera, to detect hand gesture input. Even if it did have such a sensor, the smartphone (or other device or system) lacks, for example, the ability to identify such gestures as distinct inputs and / or the appropriate instructions to respond to such gestures, even if they are recognized.

[0083] Various embodiments support functionality that addresses at least two challenges: identifying free-space gestures and reporting those gestures so that they can be used as input. While these do not necessarily represent the only features or advantages, the examples detailed below address both challenges.

[0084] Addressing the above-mentioned problems to enable touchscreen input devices and systems to accept free-space gestures provides various advantages. For example, writing a new operating system for an electronic device that accepts free-space gesture input and / or modifying an operating system to enable free-space gesture input can present significant challenges. Creating an operating system and / or making major modifications to an operating system is typically known to be expensive, time-consuming, and potentially buggy. Similarly, writing or adapting individual programs to accept free-space input is also problematic. According to various embodiments, existing code, operating systems, programs, libraries, plug-ins, and existing hardware can be used with free-space input. This saves time, effort, money, etc., and also simplifies the continued use of much of the existing software (i.e., "backward compatibility").

[0085] Furthermore, users may more easily adapt to using free-space gestures if they use free-space gestures that are similar or otherwise correspond in some way to familiar touchscreen inputs. As a more specific example, users may more easily adapt to free-space gestures as inputs if a left-to-right swipe gesture in free space is made to represent a left-to-right swipe on a touchscreen with a similar or equivalent physical action and / or a similar or equivalent system response thereto. From the user's perspective, the user can "do the same thing" (at least as far as the user is concerned) and control a device or system with free-space gestures just as they control a device or system with touchscreen inputs. In fact, the way inputs are handled is quite different between touchscreen inputs and free-space gestures, but as far as the user is concerned, the obvious similarities may reduce the barrier to users adopting free-space gesture inputs.

[0086] As mentioned above, various embodiments provide functionality including, but not limited to, making free-space gestures identifiable and invoking existing functionality in response to gestures, the former functionality being described below and the latter functionality being described in more detail below.

[0087] As discussed above with reference to FIGS. 2A-2D , surface-only input approaches, such as touchscreens, typically rely on sensing events corresponding to actual contact between a finger and a physical touchpad surface. Thus, recognizing a right swipe can include recording a touch-down event and contact motion with the surface (and possibly other events, such as lift-off). Generally, for free-space inputs, such as hand poses and gestures, no physical surface exists. Thus, recognizing a free-space right swipe gesture as an input corresponding to a touchscreen right swipe gesture can be difficult because certain events, particularly events involving physical contact, do not and typically cannot occur in free-space gestures.

[0088] More colloquially, the problem can be thought of as asking: When people are not touching the touchscreen, would they replace a touchscreen swipe with a clearly similar free-space swipe?

[0089] Several example approaches for identifying free-space gesture inputs that can function similarly to touchscreen inputs are illustrated and described below. Note that these are merely examples. The right swipe input used above as an example is similarly presented here as an example, but the input is not limited to a right swipe, a swipe, or any other particular gesture or input. Also, approaches other than those illustrated and described herein may be equally suitable. Furthermore, while it may be useful in at least some examples to use free-space gestures that have visual similarities to surface-only inputs (e.g., free-space swipe gestures that resemble touchscreen swipes), it is not necessary. The free-space gesture need not necessarily be identical to or even similar to the surface-only input.

[0090] Referring now to Figure 3, touchscreens and other surface-limited inputs can rely on rigid shapes to support input for free-space gestures, although such shapes do not necessarily exist. For example, in the configuration of Figure 3, there is a hand 304 that has performed a left-to-right movement as indicated by movement arrow 306. If the touchscreen senses movement through physical contact with the surface, Figure 3 also shows an imaging device 310 having a field of view 312 that encompasses at least a portion of hand 304 and its movement 306.

[0091] 4A, while FIG. 3 can be considered somewhat abstract (e.g., showing a simple, disconnected camera outline), FIG. 4A is a perspective view illustrating the configuration of a device 414A in the form of a head-mounted display resembling glasses. As shown, device 414A includes, for example, a hand 404A and two image capture devices 410A that sense the hand's movement. (However, in use, the head-mounted display 414A, which may typically be worn on a person, head, face, etc., is not necessarily directly involved in gesture input, and sensing gesture input, processing and executing commands, etc., are omitted for simplicity. Note that while hand 404A is shown in FIG. 4A and is also shown and referred to elsewhere for clarity, neither the hand nor the user as a whole should necessarily be considered part of the embodiment itself.)

[0092] Figure 4B shows a similar configuration to that of Figure 4A, but from a different perspective to bring it closer to the viewer's point of view: device 414B is shown from behind with imaging device 410B positioned so that it can sense hand 404B and / or its movement.

[0093] Figure 5 shows a hand 504 in free space. The configuration of Figure 5 can be considered to represent the starting position for a particular free-space gesture input, as illustrated and described below. The configuration shown in Figure 5 can also be considered to illustrate a problem previously described: for free-space gestures, recognizing events and / or inputs using physical contact with a touchscreen or the like is problematic when such a surface is not present.

[0094] Collectively, Figures 6-10D illustrate several configurations for free-space gesture input comparable to surface-limited input. Unless otherwise indicated, the gestures illustrated in Figures 6-10D can be considered to illustrate post-gesture or in-gesture structures, where the gesture begins with a structure similar to that illustrated in Figure 5.

[0095] Referring to Figure 6, a hand 604 is performing a gesture as indicated by motion marker 606. As can be seen from motion marker 606, the hand 604 (from an initial configuration similar to that of Figure 5) first moves downward, then from left to right (from the perspective of the person performing the motion), then moves horizontally from left to right, and then moves upward while moving left to right. Figure 6 shows the motion of hand 604 as a substantially single continuous motion, as can be seen from the single continuous motion marker 606.

[0096] 6 reflects one of the problems addressed by various embodiments: the lack of a clear "cue" for free-space gestures, unlike the concrete physical contact events (e.g., touchdowns) available in surface-only input. Various embodiments can address this lack while still enabling input by using free-space gestures that are at least somewhat comparable in some respects to touchscreen input.

[0097] Several variations on the characterization, interpretation, identification, etc., of various free-space inputs are described herein. For identification purposes, the configuration shown in FIG. 6 can be considered to represent a “continuous implicit” approach. That is, free-space gestures (or other inputs) are identified, and the distinguishing feature of free-space gestures from other input gestures is that the action is considered a continuous entity (rather than broken into parts, as described below). With respect to “implicit,” in configurations in which free-space gestures are analogized to surface-only inputs, the existence of certain events associated with surface-only inputs—e.g., touching down on the surface of a touchscreen, swiping along the surface, lifting off the surface, etc.—is in some sense implicit. That is, those events may not exist in actual form (e.g., because there is no surface to touch), and no obvious substitutes may exist. With the configuration shown in FIG. 6, for example, a motion can be recognized as a swipe due to the parameters of the motion itself and / or similar elements, even though there is not necessarily an analog performed or sensed to specifically represent a touch, etc. In some sense, an event such as a touchdown can be considered implicit because the action itself defines a swipe. However, this is merely a descriptive and illustrative example, and the phrase "continuously implicit" should not be construed as limiting.

[0098] For the configuration of Figure 6, where a free-space gesture is represented as a single, substantially continuous motion, the criteria for determining whether the motion should be considered an input similar to a touchscreen swipe can be defined by the characteristics of the motion itself. For example, a free-space motion can be considered to represent a swipe gesture only if the left and right motions are substantially continuous, e.g., left and right motions without pauses. There can be a requirement for an initial downward motion, a requirement for a final upward motion, and / or a requirement that there is substantially no vertical motion in the middle of the motion. An acceleration range can also be specified. Any or all of the requirements defined above can be part of the criteria for considering the configuration of Figure 6 as an example.

[0099] Embodiments are not particularly limited with respect to what characteristics of free-space manipulation are required, excluded, and / or otherwise considered when identifying gestures, poses, and / or other inputs. While relatively simple characteristics (such as speed, direction, acceleration, etc.) are described above, other and / or more complex factors may be considered.

[0100] For example, human motions typically fall preferentially within, and in some cases are specifically limited to, particular arcs and other paths. This may result from several factors, such as joint structure. Further considering a left-right swipe as a simple example, a gesture performed by moving the hand at the wrist may have a different path, speed, range of motion, etc., than a gesture performed by moving the arm at the elbow. Based on these differences, gestures can be distinguished to determine whether a gesture is intentional as opposed to accidental. The free-space input criteria may consider the above factors (however, this is merely an example and is not required).

[0101] Additionally, criteria can include positive as well as negative elements, i.e., exclusions and requirements. For example, elements that exclude certain actions from being interpreted as a particular free-space gesture input can also be part of the criteria. As a more concrete example, consider two gestures: swipe and tap.

[0102] As discussed above, Figure 6 illustrates an example of a free-space swipe gesture. Next, Figure 7 illustrates an example of a free-space tap gesture. In certain cases, a tap can be considered a generally vertical movement, e.g., a downward movement with an abrupt stop and / or abrupt reverse movement. However, in the tap illustrated in Figure 7, it can be seen that hand 704 has moved from left to right (from the viewer's perspective), as indicated by movement arrow 706.

[0103] It should be noted that in practice, free-space gestures can be prone to motion "drift." That is, a person performing a gesture in free space cannot achieve the same precision, control, repeatability, etc., as input provided to a touchscreen. It can be assumed that such drift occurs, at least in part, because the user lacks a clear physical object to interact with. Physical interface devices, such as touchscreens, can provide tactile feedback (e.g., a surface to tap on) or a specific visible target to interact with. This lack of focus can lead to what is perceived as a certain "sloppiness" in providing gesture input. The configuration shown in FIG. 7 is an example of this, in which the user intends to perform a tap input but does not constrain the left-right movement of the hand 704 while doing so.

[0104] Regardless of the cause, the drift, sloppiness, and so on described above can serve to obscure gesture input. A comparison of Figures 6 and 7 shows that in both examples, hands 604 and 704 are performing left-right movements combined with an initial downward movement and a final upward movement. Criteria for free-space input based on a sequence of left-right, up, and down movements may not be able to distinguish between the swipe in Figure 6 and the tap in Figure 7.

[0105] However, as noted above, the criteria for free-space input can incorporate positive as well as negative conditions. For example, a tap can typically include a "bounce," an abrupt stop, and / or a reversal of downward to upward motion. The criteria for a swipe can be defined to rule out such a bounce, for example, by ruling out abrupt stops in vertical motion, abrupt reversals in vertical velocity, excessive vertical acceleration (equivalent to abrupt reversals), or other considerations.

[0106] Similarly, FIG. 8 is a side view illustrating another example form of free-space tap input, in which the outstretched index finger of a hand 804 moves along a downward arc as indicated by motion arrow 806. Motions such as those shown in FIG. 8 are typical of certain taps but are not typically associated with swipes. Thus, the criteria for free-space swipe input may exclude and / or limit some motion of the first joint of the index finger. Thus, if motions such as those shown in FIG. 8 are present, the motion may be considered not to be a swipe (conversely, such motions may be considered required or optional features of the criteria defining a free-space tap gesture).

[0107] More broadly, absolute motions and / or positions (and / or characteristics such as velocity, acceleration) (e.g., of various joints) and anatomical motions and / or positions can be considered either as positive elements whose presence is required by the criterion and / or as negative elements whose absence is required by the criterion.

[0108] Additionally, where, when, and / or how a motion or other elements thereof are presented may also form part of the criteria for free-space input. As one example, for ergonomic reasons, a swipe gesture may typically be performed with fingers pointing away from the user's body rather than toward the user's body. Thus, the free-space swipe criteria, at least in certain embodiments, may exclude inward-pointing finger configurations, e.g., qualifying such motions as errors or "noise," unrelated gestures by another person, intentional intervention, etc.

[0109] Furthermore, with respect to noise, the criteria can also consider the degree of various factors. For example, even if the free-space swipe criteria excludes intermediate vertical motion (e.g., to avoid misinterpreting a sloppy tap gesture as a swipe), the exclusion can include some degree of acceptable vertical motion. That is, a certain amount of vertical motion can be expected to be accidentally present due to imperfect motor control by the gesturer, imperfect position sensing of the hand (e.g., errors or variations introduced in a sensor or processor), etc. Thus, while vertical motion can nominally be excluded from the free-space swipe criteria (or along parts thereof), this exclusion need not necessarily be absolute. Colloquially, the criteria can include a "band of interpretation" for the positive and / or negative factors.

[0110] Referring now to FIG. 9 , as discussed above, criteria can apply different requirements and / or exclusions to different portions of a given input. For example, a swipe gesture can allow for vertical motion at the beginning and end, but not in the middle. To facilitate this variation in criteria over time or distance, it may be useful, at least in certain embodiments, to divide the criteria and break down input considerations into individual blocks. FIG. 9 illustrates such an arrangement. In FIG. 9 , a hand 904 is performing a series of actions indicated by action arrows 906A, 906B, and 906C. As can be seen from a comparison with FIG. 6 , the combination of actions represented by action arrows 906A, 906B, and 906C in FIG. 9 is at least somewhat similar to the action indicated by action marker 606 in FIG. 6 .

[0111] However, in the configuration of Figure 9, the overall movement is further divided into a downward and rightward movement (from the user's perspective) 906A, a subsequent horizontal movement to the right 906B, and another subsequent upward and rightward movement 906C. In the example of Figure 9, the movements can be categorized into specific characteristics, for example, a downward vertical movement 906A, a lack of vertical movement 906B, and an upward vertical movement 906C.

[0112] The above classifications are within the scope of various embodiments. A criterion may be subdivided into two or more sub-criteria, e.g., in the configuration of FIG. 9, one sub-criterion is represented by each of action arrows 906A, 906B, and 906C. Sub-criteria may also be considered independent, self-contained entities. For example, in certain embodiments, action arrows 906A, 906B, and 906C may each represent a single "state," e.g., a downward movement with certain parameters, a horizontal movement with other parameters, and an upward movement with yet other parameters.

[0113] While this subdivision of free-space inputs into states may be merely a logical or analytical approach in some cases, defining free-space gesture actions in terms of states can be a feature of the invention itself, at least in certain embodiments. For example, state collections, state placement rules, and the like can be implemented as a form of programming rules. Thus, the basis for a particular free-space gesture may be a sequence of states; for example, if states are identified by capital letters, the gesture may be represented as ACBA, indicating a particular four-state sequence consisting of A, B, and C states (e.g., different rules are used for states that are executed in parallel, such as multi-finger gestures, two-handed gestures, etc.).

[0114] As a more specific example, if state A is a downward movement, state B is a sudden inversion, and state C is an upward movement, then the tap gesture can be ABC, and the double tap can be ABCABC. Note that states do not necessarily have to be specific to one gesture; states A and C in the tap / double tap example (downward and upward movements) can also be used as part of a swipe gesture, e.g., ADC. However, D is a horizontal movement (left to right, right to left, etc.).

[0115] Indeed, a set of states may resemble a "language" in certain embodiments. A relatively small set of states can be used to define a very large set of gestures, where each state (as shown in FIG. 9 ) represents a distinctly defined action, structure, and / or other input element (such as a pause without movement, a voice input, a finger snap or finger-to-surface noise, a keystroke, etc.). For example, if a set of 25 states is arranged in a three-state gesture, over 15,000 gestures are mathematically possible. For a given number of states, as the number of states per gesture increases (e.g., from three states to four states in the example above), the number of mathematically possible gestures also increases. The number of gestures can also be increased if the number of states per gesture is not fixed (e.g., the gesture includes at least one to six states).

[0116] In practice, not all of the above gestures will be convenient for the user, not all will be easily distinguishable by available sensors, etc., and not all mathematically possible gestures will be available for a given set of states (even in embodiments that do utilize a state language, although this is not required, not all possible gestures need be used). However, even just 1% of the approximately 15,000 gestures in the above example represents approximately 150 gestures. Thus, even a relatively small number of states can support a broad and / or diverse set of gesture inputs.

[0117] Building a gesture language using states in this manner can provide certain advantages, for example, in terms of efficiency. In the example above, 150 unique gestures can be assembled from only 25 states. Thus, instead of defining, coding, and debugging 150 different gestures, it would be more appropriate to define, code, and debug 25 states. Having fewer states than gestures reduces resource costs, at least in certain cases. Also, gestures that consist of multiple states are typically longer and / or more complex than the individual states that make up the gesture. Thus, defining, coding, and debugging individual states is more resource-intensive than defining, coding, and debugging each complete gesture, further saving resources (e.g., time, money, processing power, memory, etc.).

[0118] However, the use of a state language, such as the use of states and / or other individual gesture elements, is merely one example and is not required; other configurations may be equally suitable.

[0119] As discussed above with reference to Figures 2A-2D, touchscreen input can be interpreted as a series of events, such as touchdown, swipe, and liftoff, in certain cases. While the configuration shown in Figure 9 can be interpreted as representing three free-space states that correspond one-to-one to the three touchscreen events of Figures 2A-2D, this is merely an example. A one-to-one correspondence with touchscreen events is not required, and touchscreen events (if used) are not necessarily considered when setting the free-space criteria (although a one-to-one correspondence with such considerations is not excluded).

[0120] For example, in contrast to the configuration shown in Figure 6, which for comparison purposes is described as a "continuous implicit" approach, the configuration of Figure 9 can be referred to as a "discrete implicit" approach. Because the free-space swipe gesture shown in Figure 9 is divided into and / or viewed as three states rather than a single continuous action, the free-space gesture that invokes a surface-limited response according to Figure 9 does not include an explicit equivalent of an event such as touching down on a surface. Such elements can be described as being inferred from, for example, actions 906A, 906B, and 906C.

[0121] Referring now to Figure 10, some embodiments may include defining a boundary for a free-space gesture. In the configuration shown in Figure 10, a hand 1004 is performing a free-space swipe input with motions indicated by motion arrows 1006A, 1006B, and 1006C (which may, but may not necessarily, represent the individual states that make up the overall motion). Also in Figure 10, a boundary 1016 is shown as a rectangular plane (with a cross-hatched border for clarity).

[0122] In the example of Figure 10, boundary 1016 serves as a proxy for a physical surface, such as a touchpad. Once boundary 1016 is defined, contact with boundary 1016 can also be defined. For example, a touchdown equivalent (shown in Figure 10 by marker 1008A) can be considered to occur when hand 1004 touches boundary 1016, passes through boundary 1016, approaches within a minimum distance of boundary 1016, etc. Similarly, a liftoff equivalent (shown in Figure 10 by marker 1008B) can be considered to occur when hand 1004 makes discontinuous contact with boundary 1016, exits boundary 1016, moves away from boundary 1016, etc., and continuous contact or proximity of hand 1004 to boundary 1016 can be considered to be motion along a surface.

[0123] The inclusion of a boundary such as that shown in FIG. 10 may be useful in at least certain embodiments. For example, defining a surface or other geometric shape—even a geometric shape without physical substance—may enhance and / or simplify certain definitions. Due to the presence of the boundary surface 1016, a free-space swipe criterion may be defined such that the user's hand 1004 must contact the boundary surface 1016; for example, any action that does not include proper contact between the hand 1004 and the boundary surface 1016 may be ignored as not constituting a free-space swipe. This may be useful, for example, to distinguish between input gestures and noise, or to distinguish between similar gestures.

[0124] Additionally, if the boundary 1016 is visible to the user, for example displayed as an augmented reality entity on a head-mounted display, the boundary 1016 can act as a visual guide as to where and / or how a gesture should be performed, assisting the user in performing free-space gestures.

[0125] 11A illustrates a first portion of another exemplary configuration for identifying a free-space swipe gesture according to various embodiments. Again, a boundary 1116A is incorporated, with a hand 1104A in an initial position above the boundary 1116A. However, as illustrated, the boundary 1116A in FIG. 11A encompasses a solid rather than being planar as in FIG. 10. The boundary can be defined in a three-dimensional sense (i.e., the entire solid is the boundary) or as a containing surface. For purposes of this description, either interpretation of FIG. 11A is acceptable.

[0126] As shown in FIG. 11B, hand 1104B is lowered into boundary 1116B, as indicated by motion arrow 1106B. Contact marker 1108B also indicates that the index finger of hand 1104B intersects the surface of boundary 1116B. The configuration shown is an example. In other embodiments, the entire hand 1104B within boundary 1116B may be considered to be in contact with the boundary (not a point of contact), the point of contact may be located on and move with the tip (or other feature) of the index finger of hand 1104B, etc., and other configurations may be equally suitable.

[0127] Continuing to refer to Figure 11C, as can be seen from motion arrow 1106C, hand 1104C is moving to the right (from the viewer's perspective) while maintaining contact point 1108C with boundary 1116C. Similarly, in Figure 11D, as can be seen from motion arrow 1106D, hand 1104D is moving to the right and upward, removing contact with boundary 1116D at contact marker 1108D.

[0128] Thus, as can be seen from Figures 10 and 11A-11D, the boundary is not limited to one particular shape. While two shapes are shown, other embodiments may vary widely and may incorporate boundaries including, but not limited to, curved or solid surfaces, multiple surfaces, other configurations, etc.

[0129] Additionally, for certain embodiments, it may be useful to position the boundary so that it is at least approximately aligned with some or all of a physical object, such as a table, desk, window, clipboard, etc. For example, aligning the boundary with a physical desktop can provide haptic feedback. That is, even if a physical touchpad is not present, the user can feel physical contact between their fingertip and the desktop that is at least somewhat similar to contact between their fingertip and a touchpad. Note that the degree of alignment between the boundary and the physical object need not be perfect or even near-perfect. For example, considering the example of a boundary that is still aligned with the desktop, a misalignment of a few millimeters or even a centimeter or more can facilitate the user's tactile interpretation that they are "using a touchscreen" (even if a physical touchscreen is not present). Whether or not the user notices the degree of alignment does not necessarily limit the degree of alignment. Even if the misalignment is detectable to the user, haptic feedback or other benefits can be obtained. As long as functionality is maintained, various embodiments do not impose strictly measured mathematical or logical alignment limits.

[0130] In fact, for at least some embodiments, misalignment (whether intentional or accidental) can itself be effective. Continuing with the example above, even if there is misalignment such that the boundary is somewhat above the desktop, the user can still hold their fingertip above the desktop to perform a "hover" action (e.g., similar to a "mouseover" for a computer mouse). However, these are merely examples, and other configurations may be equally suitable.

[0131] The configurations shown in Figures 10-11D can be referred to as "passive-explicit." That is, they can provide a near-explicit equivalent to touch-down events and similar events, for example, by defining a boundary and considering contact with the boundary to represent a touch-down. Such configurations can be considered passive because the user does not necessarily have to "do anything" to indicate contact or its equivalent. Rather, the user simply performs a free-space input and makes (or does not make) contact with the boundary.

[0132] 12A-12D, the free space criteria may include (for example) some indication that identifies a hand movement as a gesture. In the illustrated example, the indication is an additional explicit hand pose, specifically an extended pinky finger.

[0133] In Figure 12A, hand 1204A is visible before performing a free-space swipe gesture. Note that the index finger is extended, but the pinky finger is not. Proceeding to Figure 12B, hand 1204B is still shown in the initial position, but with the pinky finger extended. In this example, the pinky finger is extended as an indication of gesture input 1218B. That is, the extended pinky finger signals (e.g., to a processor evaluating sensor data) that the action performed should be interpreted as having some characteristic, e.g., a swipe, another gesture, one of a group of gestures (although in the illustrated example, the action represents a swipe).

[0134] In Figure 12C, hand 1204C is moving from left to right (from the perspective of the person controlling hand 1204C) as indicated by movement arrow 1206C, with the pinky finger still extended as shown in Figure 1218C. In Figure 12D, the movement is complete and the pinky finger is curled.

[0135] Thus, in the configurations of Figures 12A-12D, an extended pinky finger can serve as an indication related to the input criteria. The exact meaning of the indication may vary even within the illustrated examples; the indication may refer specifically to a swipe input, more generally to an input that should be interpreted as a touch on the virtual surface, an input that is considered only in a particular program or under particular circumstances, etc. Similarly, while Figures 12A-12D show an extended pinky finger, other indications may be equally appropriate. For example, to simplify the distinction between a swipe input and a tap input, the swipe gesture criteria require the index finger to be extended, while the tap gesture criteria require both the index and middle fingers to be extended. The indications are not limited to hand or finger positions, and the range of indications possible within various embodiments is very large.

[0136] Typically, but not necessarily, the display can be at least substantially contemporaneous with the movement to collectively represent the gesture. Continuing with the example shown in Figures 12A-12D, the display of the extended pinky finger is present throughout the hand movement (i.e., the finger is extended) to identify the hand movement as an input gesture. Thus, the displays in Figures 12A-12D can be said to be contemporaneous with the movement.

[0137] However, as can be seen from Figures 12A and 12D, the display (the extended pinky finger) is present even when the hand is not moving. As can be seen from Figures 12A and 12D, the display need not be fully simultaneous with the movement (or other input). That is, the display can precede and / or follow the movement, and / or the movement can precede and / or follow the display. Thus, a configuration in which the pinky finger is extended and then curled can still serve as a display; for example, the following hand movement should be considered a gesture input. Also, while for at least some embodiments it is convenient to utilize a display that is partially or fully simultaneous with the movement, this is merely an example, and other configurations may be equally suitable.

[0138] 12A-12D can be referred to as "active-explicit." Here, too, an explicit equivalent of a touch-down event and similar events can be provided in the form of an indication, such as a hand pose, that is actively provided by the user extending a finger or otherwise changing the hand pose.

[0139] Four broad categories of approaches have been described in accordance with various embodiments to support free-space input for surface-limited control. Somewhat colloquially, these four approaches—continuous implicit, discrete implicit, passive-explicit, and active-explicit—serve as tools for addressing the differences between free-space input and surface-limited input. Thus, through the use of one or more of these approaches in accordance with various embodiments, sufficient information can be obtained from the free-space input to identify and distinguish its equivalent from a touchscreen input, and free-space gestures can replace or “stand-in” for touchscreen input, at least for certain purposes. The above configurations enable, or at least contribute to, the control of devices, systems, and the like that recognize touchscreen input using free-space gestures.

[0140] Although four exemplary approaches have been described above, the approaches are not necessarily limited to these four, and other configurations for identifying and / or distinguishing free-space inputs so that surface-only inputs can be invoked may be equally suitable. Furthermore, the exemplary approaches are not necessarily exclusive. For example, some free-space gestures may be addressed using a continuous implicit approach, while other free-space gestures (even within the same device, system, etc.) may be addressed using a discrete implicit approach. Similarly, a particular free-space gesture may be addressed using one or both of two or more approaches. For example, for a given embodiment, either crossing a planar boundary or performing a hand pose may be acceptable.

[0141] It should be noted that, in at least certain examples presented herein, each free-space input, such as a right swipe gesture, may be referred to as having a one-to-one correspondence with a surface-bound input and / or its response. However, this is for the sake of simplicity. Embodiments do not require a one-to-one correspondence between free-space inputs, surface-bound inputs, and / or responses. For example, two or more free-space gestures may invoke a single surface-bound input response, and other configurations may be equally suitable.

[0142] Next, Figure 13 illustrates an exemplary method for applying free-space input to surface-limited control. As discussed above, various embodiments support enabling differentiation and / or identification of free-space gestures and conveying such gestures so that they can be used as inputs. Both are illustrated schematically in Figure 13. The former has already been described in detail, with additional information provided below. The latter is also described in more detail below.

[0143] In Figure 13, free space input criteria are set in the processor (1332). As described above, the free space input criteria define what, for a given embodiment, does and / or does not constitute a particular free space input. For example, a free space swipe gesture may include movement of a hand or other end effector along a particular path within a particular range of speeds and accelerations in a particular configuration (e.g., index finger extended and other fingers curled), while excluding other elements such as movement of the first joint of the index finger. The free space input criteria are not limited with respect to criteria that may be required, excluded, or otherwise specified.

[0144] Typically, although not necessarily, free-space input criteria can correspond at least to some extent to surface-limited input, such as touchscreen input. For example, a free-space swipe can elicit a visually similar touchscreen swipe, with a similar behavior as a touchscreen swipe.

[0145] Note that "setting up" something can refer, depending on the details, to either or both creating something new (e.g., establishing a business, if starting a new business) and determining an already existing state (e.g., ascertaining the whereabouts of a person, if the location of a person already there is discovered or received from another source). Similarly, setting up free-space input criteria can encompass several possible approaches, including but not limited to:

[0146] Setting the free-space input reference in the processor may include instantiating the free-space input reference in the processor from some source, e.g., a data storage device such as a hard drive or solid-state drive, through a communication means such as a wired or wireless modem. Setting the free-space input reference may also include computationally creating the free-space input reference in the processor using executable instructions. Embodiments are not limited with respect to how the free-space input reference is set. The only requirement is that a functional free-space input reference be made available in some way. Configurations other than those described may be equally suitable. Also, setting, when used in other steps herein, should be used in a similarly broad sense.

[0147] Similarly, embodiments are not limited with respect to the nature of the free space input reference. Typically, the free space input reference may be a collection of data and / or executable instructions instantiated in a processor, such as a file, a program, or portions thereof. However, this is merely an example, and other configurations may be equally suitable.

[0148] Also, embodiments are not limited with respect to the processor for which the free-space input reference is established. A variety of general-purpose, purpose-specific, and embedded systems may be suitable for use as a processor. Furthermore, a processor may equally well include two or more physical or logical processors and / or processor components, or be a "virtual" processor. Other configurations may equally well be suitable.

[0149] The above-described functionality for identifying and / or distinguishing free space gestures can be considered, in a sense, to be encapsulated in step 1332. While the determination of whether a particular phenomenon (gesture, pose, etc.) represents a free space input, i.e., whether the free space input criteria are met, is addressed sequentially in Figure 13, the input can, in a sense, be defined or generated in step 1332. Thus, the specific discussion with reference to Figures 5-12D can be considered particularly relevant to step 1332.

[0150] Continuing with reference to FIG. 13 , free-space input is sensed 1334 using a sensor. Embodiments are not limited with respect to what sensor is utilized to sense 1334 the free-space input, nor with respect to how the free-space input may be sensed 1334. Typically, but not necessarily, some form of imaging device, such as a camera, depth camera, or the like, may be utilized to capture one or more images and / or video segments to determine position, motion, etc., indicative of the free-space input. While the use of cameras, including but not limited to CMOS and CCD cameras, is suitable for certain embodiments, other configurations may be equally suitable. Additionally, while step 1334 refers to a single sensor, the use of multiple sensors, similar or different (e.g., a stereo pair of imaging devices), operating individually or in concert, may be equally suitable.

[0151] The free-space input is communicated to the processor (1336). Embodiments are not limited with respect to the content and / or format of the communication, e.g., image, video, numerical data, etc. Nor are embodiments limited with respect to how the free-space input is communicated. Typically, but not necessarily, the sensor can be located on the same device or system as the processor, and communication can occur via a wired link. However, other configurations, including but not limited to wireless communication (whether the sensor and processor are located on the same device or system, nearby, or remotely), may be equally suitable.

[0152] In the processor, it is determined 1338 whether the free space input (sensed in step 1334 and communicated to the processor in step 1336) meets the criteria (set in step 1332). Typically, although not necessarily, this determination may have the form of a comparison performed by executable instructions instantiated in the processor, although other configurations may be equally suitable.

[0153] If decision 1338 is positive, i.e., the free space input meets the criteria, the method proceeds to step 1340. If decision 1338 is negative, i.e., the free space input does not meet the criteria, step 1340 is skipped.

[0154] Continuing at step 1340, a surface-only input response is invoked. That is, some function or action that can typically be associated with a surface-only input, such as a touchscreen input, is performed. This can be considered, in some sense, to represent the functionality described above for signaling and / or interpreting free-space gestures so that they are available as surface-only input. Approaches for doing so are described in further detail later herein. However, embodiments are not limited as to how a surface-only input response can be invoked. Suitable approaches include, but are not limited to, "spoofing" or generating virtual or "pseudo" signals that represent real touchscreen input, generating virtual signals that confirm that touchscreen input has been received, or directly initiating or executing a command associated with the touchscreen input.

[0155] Although FIG. 13 shows the method as completing after step 1340 for simplicity, in practice the method may be iterative, e.g., looping back to step 1334 to continually sense free-space input.

[0156] Additionally, Figure 14 illustrates another exemplary method for applying free-space input to surface-only control. The configuration shown in Figure 14 more specifically addresses the example of using the continuous implicit approach (shown and described above).

[0157] In Figure 14, the free space input criteria are instantiated 1432 into the processor, for example, by loading the free space input criteria into the processor from a data storage device such as a solid state drive. Such instantiation can be considered an example of setting the criteria as described above with reference to Figure 13, for example.

[0158] Instantiating the free-space reference (1432) can be viewed as several sub-steps and / or exhibiting several features identified as 1432A-1432F in FIG. 14. The details are merely exemplary, and more specifically, provide an example of a continuous-implicit approach, i.e., an approach in which a free-space input is defined as (and / or subsequently treated as) a single, approximately continuous action, without necessarily including an explicit "proxy" for features that can be associated with a touchscreen input (or other surface-bound input), such as a touchdown.

[0159] For illustrative purposes, the free-space input criteria instantiated 1432 in the processor in Figure 14 (as well as Figures 15, 17, and 18) is represented as a right-swipe motion in free space, similar to certain other free-space inputs previously described and illustrated herein, for example. Embodiments are not limited to right-swipe, and other configurations may be suitable.

[0160] In the example configuration of Figure 14, the free-space input criteria instantiated 1432 in the processor may include requiring an extended index finger to act as an end effector to provide the free-space input 1432A. Thus, for the specific example of Figure 14, use of a stylus or other end effector is not recognized as meeting the free-space input criteria, and therefore, any input not provided with an extended index finger is not accepted, even if the free-space input criteria are otherwise met.

[0161] Embodiments are not limited to defining free-space input as being limited to input provided by an extended index finger as an end effector. For at least certain embodiments, it may be useful to acquire one or more images or video clips and identify the presence (or absence) of an extended index finger through image recognition. Contours of the fingers and / or hand or a hand model (e.g., an articulated model, such as "joints" or rigid "bones") may be utilized as part of the image recognition and / or independently. The trajectory of the end effector's motion through multiple images and / or videos may also be considered. Stereo sensing, time-of-flight sensing, focal depth sensing, and / or other approaches incorporating depth and / or distance as three dimensions may also be considered. These are merely examples, and other configurations may be equally suitable. Numerous approaches may be utilized to determine whether a free-space input is provided by an index finger or other particular end effector, and embodiments are not limited with respect to which approaches may be utilized.

[0162] More broadly, subsystem 1432A can be implemented in a variety of ways, and embodiments are not limited to how step 1432A is performed. Similarly, steps 1432B-1432F and other steps in Figure 14 are presented as examples and should not be construed as limiting embodiments to specific implementations and / or methods of implementation unless otherwise specified.

[0163] 14, instantiating 1432 the free space input criteria in the processor can include excluding 1432B extension of other fingers. That is, if fingers other than the index finger are extended, the free space input criteria cannot be met, and therefore input provided with an extended finger other than the index finger is not accepted, even if the free space input criteria are otherwise met.

[0164] Taken together, 1432A and 1432B can be understood to require only the index finger to be extended to perform a free-space input and meet the free-space input criteria. As noted above, the example in FIG. 14 relates to a right-swipe free-space input. Taken together, 1432A and 1432B can be useful, for example, in identifying a hand motion as a free-space input (e.g., by being performed with an extended index finger) and / or in distinguishing a right-swipe input from other free-space inputs (e.g., other gestures provided with two fingers) and / or accidental motions.

[0165] The free-space input criteria also specify a left-right displacement range (1432C): that is, the total variation in index finger displacement from left to right must fall within a particular range, e.g., less than a maximum displacement and more than a minimum displacement. Such a range can help define a right swipe and / or distinguish it from other inputs and / or accidental movements, since a right swipe can be assumed, at least in certain circumstances, to exhibit the smallest and / or largest "size" with respect to the overall left-right movement.

[0166] The free space input criteria further specify (1432D) that the motion (e.g., of an extended index finger) exhibits continuous downward, left-right, and upward motion within a particular range (e.g., within a minimum and maximum motion, within a particular path shape, etc.). The above specification (1432D) can be useful, for example, in distinguishing a particular motion, such as a right swipe in the example of FIG. 14, by the expected characteristic motion.

[0167] With further reference to FIG. 14 , the free-space input criteria limits rebound acceleration (1432E). As noted above, certain free-space inputs, such as taps, may resemble right swipes, at least under some circumstances. More broadly, free-space inputs may resemble one another, especially when those inputs deviate from ideal shapes and / or are distorted (e.g., when a user is “sloppy” in providing the input). As noted above, at least under certain circumstances, individual features, in this case rebound acceleration, can be used to distinguish free-space inputs that appear similar despite some distortion. For example, taps include an acceleration profile that can be referred to as rebound (a downward motion that rapidly reverses from an upward motion), while swipes do not include such acceleration. Thus, limiting rebound acceleration to a maximum (1432E) can filter out taps (including, but not limited to, distorted taps that include a left-right motion similar to a right swipe). In other words, considering acceleration can distinguish between swipes and non-swipe inputs and / or accidental motions.

[0168] It should be noted that in certain embodiments, it may be useful to completely exclude rebound acceleration or other possible characteristic motions, accelerations, or other features. However, while such absolute exclusion is not prohibited, it is beneficial in at least some embodiments to allow for at least some characteristic features to accommodate certain forms of noise. For example, in some cases, human hand motion is imperfect (e.g., tremors, wind, motion variations due to clothing restrictions or injury, etc.), motion sensing by sensors is imperfect, or motion interpretation by a processor based on sensor input is imperfect.

[0169] The above limitations can, in certain embodiments, be considered (and implemented with) a form of “low-pass filtering” of the motion. For example, motions below a certain velocity, a certain displacement, and / or a certain acceleration can be excluded from consideration. The above configuration can exclude small “wobble” or other accidental motions of the fingertip. The low-pass filter (if any) can be implemented within the sensor or processor and, quite strictly, cannot be part of the free-space input criteria itself. However, functionally, low-pass filtering the motion to eliminate accidental wobble of the fingertip tip, or low-pass filtering the acceleration (as in the example above) to eliminate accelerations that are too low to represent part of a tap gesture, can function similarly to including a limit on the motion and / or rebound acceleration (and / or other feature).

[0170] Continuing with FIG. 14 , the free-space input criteria limits the range of finger joint motion (1432F). As discussed above, certain inputs and / or contingent actions other than right swipes may involve finger joint motion. For example, a tap or touch may be performed by moving the index finger with the proximal knuckle. Similar to limiting rebound acceleration (1432E) above, limiting finger joint motion (1432F) can also be useful, for example, in identifying swipe inputs and / or filtering out other inputs and / or non-input actions.

[0171] Collectively, elements 1432A-1432F can be considered to represent free space input criteria that are instantiated 1432 in the processor. Although the free space input criteria are shown in FIG. 14 as including multiple independent elements (e.g., elements 1432A-1432F) of various types (e.g., requirements, ranges, exclusions, etc.), embodiments are not limited to defining the free space input in this manner, nor to the various example elements 1432A-1432F shown.

[0172] For example, it may be appropriate to require that there be no pauses and / or pauses in the gesture to avoid confusing two consecutive gestures with a pause between them with one long gesture. Similarly, it may be appropriate to require that the movement fall within an anatomically and / or behaviorally preferred arc of movement. Human movements may typically follow particular arcs and / or other paths depending on factors such as the physical structure of a person's joints, cultural preferences for movement, and personal habits. Only inputs that follow the above parameters may be considered, and non-input movements may be excluded from consideration as gestures.

[0173] Also, for simplicity, the various elements referred to in substeps 1432A-1432F are described as fixed, but embodiments are not limited to fixed free-space input criteria (or portions thereof). The criteria may be changeable based on user selection, may accommodate and / or adjust to changing conditions and / or changing gestures, and / or may be otherwise changed.

[0174] Other configurations may be equally suitable.

[0175] 14, a free space input is sensed using a sensor (1434). The free space input is communicated to a processor (1436). The processor determines (1438) whether the free space input meets a criterion. If the determination 1438 is positive (i.e., the free space input meets the criterion), the method proceeds to step 1440. If the determination 1438 is negative (i.e., the free space input does not meet the criterion), step 1440 is skipped. Continuing in step 1440, a surface-limited input response is invoked.

[0176] 15A and 15B illustrate another example method of applying free-space input to surface-limited control. The configuration shown in Figures 15 and 15B more specifically addresses an example of the discrete implicit approach (illustrated and described herein).

[0177] 15A, the free space input criteria are instantiated 1532 into the processor, for example, by loading the free space input criteria into the processor from a data storage device such as a solid state drive. Such instantiation can be considered an example of setting the criteria as described herein above.

[0178] As noted above, the configuration of FIG. 15A addresses a discrete implicit approach, i.e., an approach considered in two or more separate parts. For illustrative purposes, in FIG. 15A , a free-space swipe input is considered in three parts: touch-down, swipe, and lift-off. Similarly, the free-space input criteria are considered in three parts, and instantiating the free-space criteria in the processor 1532 can be considered as three separate (but at least interrelated) substeps. For illustrative purposes, in FIG. 15A , those three substeps are referred to as instantiating the free-space touch-down state subcriterion in the processor (1532A), instantiating the free-space swipe state subcriterion in the processor (1532B), and instantiating the free-space lift-off state subcriterion in the processor (1532C).

[0179] Each of the sub-criteria instantiated in the processor in steps 1532A, 1532B, 1532C can then be considered as several sub-steps.

[0180] Referring to subsystem 1532A, instantiating the free-space touchdown condition subcriterion in the processor includes requiring 1532A1 the extended index finger to act as the end effector providing the free-space input and excluding 1532A2 the extension of the other fingers.

[0181] The free space touchdown condition subcriterion also defines a range of downward motion (e.g., minimum and / or maximum distance of downward motion for an extended index finger) (1532A3).

[0182] Still referring to FIG. 15A, the free space touchdown condition subcriterion limits the rebound acceleration (1532A4) and also limits the range of motion of the knuckles (1532A5).

[0183] Collectively, elements 1532A1-1532A5 can be considered to represent 1532A free-space condition subcriteria (specifically, touch-down conditions) instantiated in a processor. As noted above, although the free-space touch-state subcriteria are shown in FIG. 15A as including multiple independent elements (e.g., elements 1532A1-1532A5) of various types (e.g., requirements, ranges, exclusions, etc.), embodiments are not limited to defining free-space inputs in this manner, nor to the various illustrative elements 1532A1-1532A5 shown. Other configurations may be equally suitable.

[0184] It may be useful to point out a contrast between the configurations of FIGS. 14 and 15A. While the configuration of FIG. 14 defines a full range of left-right motion (1432C) and considers the sequence of down, left-right, and up motions as a single continuous input, the configuration of FIG. 15A treats each of the three motions (down, left-right, and up) as individual states. Thus, while the substeps of 1532A (and also 1532B and 1532C, described below) are slightly different, the overall motion and input are similar. Even if the overall motions by which a user generates an input (a swipe motion) are similar or identical in the configurations shown in FIGS. 14 and 15A, the interpretation of those motions may differ between different embodiments. With more specific reference to FIGS. 14 and 15A, the configuration of FIG. 14 considers free-space input criteria in terms of a continuous motion, while the configuration of FIG. 15A considers free-space input criteria in terms of three distinct states.

[0185] Continuing with reference to FIG. 15A (further step 1532) and sub-step 1532B, instantiating the free-space touchdown condition sub-criterion in the processor includes requiring the extended index finger to act as the end-effector providing the free-space input (1532B1) and excluding extension of the other fingers (1532B2).

[0186] The free space touchdown condition subcriterion defines a range of side-to-side motion (1532B3), limits rebound acceleration (1532B4), and also limits the range of motion of the finger joints (1532B5).

[0187] 15A, instantiating the free-space touchdown condition subcriteria in the processor includes requiring the extended index finger to act as the end-effector providing the free-space input (1532C1) and excluding extension of the other fingers (1532C2). The free-space touchdown condition subcriteria prescribe a range of upward motion (1532C3), limit the rebound acceleration (1532C4), and also limit the range of motion of the finger joints (1532C5).

[0188] Thus, as shown in FIG. 15A, instantiating 1532 a free space input criterion can be viewed as instantiating three free space state sub-criteria 1532A, 1532B, 1532C for three distinct states.

[0189] Also, as shown in FIG. 15A , instantiated sub-criteria 1532A, 1532B, and 1532C have at least some similarity. For example, 1532A, 1532B, and 1532C all require an extended index finger and limit recoil acceleration. This may be useful, at least for certain embodiments, for example, in terms of input consistency. However, this is merely an example; different states (or other input criteria and / or individual portions of the input) need not involve parallel sub-steps or otherwise resemble one another; other configurations may be equally suitable.

[0190] Next, referring to Figure 15B, a free space input is sensed using a sensor (1534). The free space input is reported to a processor (1536). In the processor, a determination is made as to whether the free space input meets a criterion (1538). If determination 1538 is positive, the method proceeds to step 1540; if determination 1538 is negative, step 1540 is skipped. Continuing in step 1540, a surface-limited input response is invoked.

[0191] Turning to FIG. 16 with reference to FIGS. 14 and 15, free-space input criteria and characteristics and state sub-criteria and characteristics are shown. The examples in FIGS. 14 and 15 mentioned characteristics such as an extended index finger, specific motions and / or ranges of motion, and acceleration limitations. FIG. 16 shows examples of how inputs can be sensed and / or how it can be determined whether the above characteristics are met (e.g., steps 1534 and 1538 of FIG. 15B).

[0192] Figure 16 shows a sequence 1605 of hands with motion arrows 1606 indicating the direction of motion. The configuration in Figure 16 shows the position and structure of a hand (or other end effector) as it moves through space and / or the change in structure over time, with each hand in sequence 1605 representing the hand at each instant in time during the period (for simplicity, the structure of the hand does not change in Figure 16, e.g., the position of the fingers relative to the hand; in practice, changes may occur, and embodiments are not limited to configurations where the structure does not change).

[0193] A sequence 1605 such as that shown in FIG. 16 can be obtained in various ways by capturing a series of images over a period of time, with each hand in the sequence 1605 representing the position and / or structure of the hand in a separate image, and each image representing a moment in time during the period. Typically, but not necessarily, such a sequence 1605 can be obtained using an RGB camera, a depth camera, or other sensor, and the images and / or data derived from the images are communicated to and / or evaluated by a processor. However, it should be noted that the configuration of FIG. 16 is merely an example. Such visual configurations may or may not represent any point in a particular embodiment. The images can be considered individually, rather than arranged as shown in FIG. 16. Alternatively, wireframes, skeletons, articulated models, or other structures (possibly, but not necessarily, derived from the images) can be utilized in addition to or instead of the images. As yet another option, the images can be considered not graphically but instead numerically or in other formats.

[0194] Regardless of how such a sequence 1605 is obtained and / or examined, it can be understood that evaluating a sequence 1605 such as that shown in FIG. 16 can determine characteristics such as, but not limited to, position, range of motion, velocity, acceleration, and joint position variance, for example, as described above in connection with step 1432 of FIG. 14 and 1532 of FIG. 15 .

[0195] In the configuration shown in Figure 16, the actions displayed are considered together (as indicated by the single action arrow 1606), e.g., using a continuous implicit approach, as a single input. However, referring now to Figure 17, sequence 1706 can be considered as multiple actions, inputs, states, etc., as indicated by action arrows 1706A, 1706B, and 1706C. With the configuration shown in Figure 17, action arrow 170A can represent a touch-down state indicated by some or all of sequence 1705, action arrow 1706B can represent a swipe state indicated by some or all of sequence 1705, and action arrow 1706C can represent a lift-off state indicated by some or all of sequence 1705. Thus, the configuration of Figure 17 can be considered to represent a discrete implicit approach.

[0196] Note that sequence 1705 does not necessarily need to be subdivided itself to identify states that may represent only a portion of sequence 1705. That is, the entire sequence 1705 can be evaluated when identifying each state (touch down, swipe, lift off), or only a portion of each state can be evaluated (e.g., approximately one-third of the way down the right side of the page for touch down).

[0197] It will be appreciated from Figures 16 and 17 that similar configurations are applicable to passive explicit, active explicit, and / or other approaches. A passive explicit configuration may be similar to the configuration shown in Figures 16 and / or 17 (where the boundary is not visible, as it may only exist as a virtual or informational structure), while an active explicit configuration may further include, for example, an extension or other indication of the pinky finger (as already shown in Figures 12A-12D).

[0198] Next, FIG. 18 illustrates another exemplary method for applying free-space input to surface-limited control using a passive-explicit approach (already shown and described herein).

[0199] In FIG. 18, a free space input reference is instantiated in the processor (1832).

[0200] Instantiating a free-space reference in the processor (1832) can be viewed as several substeps. A finite, planar boundary is defined (1832A). The boundary serves as an explicit "surrogate" for a touchscreen or other confining surface, allowing a user to provide input in free space without using such a confining surface and without actively modifying (hence "passive") said input. Typically, but not necessarily, the boundary can be an imaginary structure defined in space (by the processor), such as a virtual or augmented reality structure, that can be displayed to a person providing free-space input. However, the boundary need not be displayed; an invisible structure defined in space (e.g., a region not visually output but mathematically defined in space within the processor) may be equally suitable.

[0201] For purposes of FIG. 18, the boundary is rectangular and planar (i.e., approximately two-dimensional), like the boundary shown in FIG. 10. Such boundaries are convenient because of their similarity to, for example, physical touchscreens familiar to users. However, a rectangular and planar boundary is merely an example, and other configurations may be equally suitable. As discussed above with reference to FIG. 10 and FIGS. 11A-11D, the boundary can take on a variety of shapes, including, but not limited to, a plane or flat surface, a rectangular solid surface and / or solid, a sphere or portion thereof, other curved surfaces, and any surface or solid.

[0202] Continuing with FIG. 18, instantiating the free space input criteria in the processor includes requiring touchdown between the extended index finger and the boundary (1832B). That is, the extended index finger must contact the boundary (or, in the case of an intangible boundary, must coincide with and / or pass through the boundary). This can be understood as the explicit free space equivalent of a touchscreen touch event.

[0203] Instantiating the free space input criteria in the processor further includes requiring a left-right movement of the index finger while in contact with the boundary (1832C) and requiring a lift-off that terminates contact of the index finger with the boundary (1832D), which similarly can be understood as the explicit free space equivalents of a touchscreen swipe event and a touchscreen lift-off event, respectively.

[0204] Referring to FIG. 18 , free-space input is sensed using a sensor (1834). Note that the sensor may sense motion, position, etc., but the sensor may or may not sense a boundary. For example, if the boundary exists partially or completely as a construct within the processor, the boundary need not even be sensed as a location, extent, etc. that is already “known” by the processor. Nevertheless, some type of sensory information may typically be sensed by the sensor (1834), including, but not limited to, the position, motion, and configuration of the hand or other end effector relative to the boundary.

[0205] The free space input is reported to a processor 1836. The processor determines 1838 whether the free space input meets a criterion. If the determination 1838 is positive, the method proceeds to step 1840; if the determination 1838 is negative, step 1840 is skipped. Continuing in step 1840, a surface-limited input response is invoked.

[0206] Next, FIG. 19 illustrates another exemplary method of applying free-space input to surface-limited control using an active-explicit approach (already illustrated and described herein).

[0207] In FIG. 19, a free space input reference is instantiated in the processor (1932).

[0208] Instantiating a free-space reference in a processor (1932) can be considered to have several substeps. A display is defined (1932A). The display serves to distinguish free-space input from other free-space inputs and / or non-inputs, such as incidental gestures. In the example configuration of FIG. 19, the display is defined (1932A) as including an extended pinky finger. The display itself does not necessarily provide input (e.g., a gesture), but can be considered to enable input. That is, a free-space input can be a gesture provided with an extended index finger, and the index finger can be monitored for gestures (e.g., sense position, movement, acceleration, etc.), but the gesture cannot be accepted as input unless the display is present (or at least not accepted as a specific input; for example, it may not be accepted as a swipe but may be accepted as a tap). Thus, input and display can be distinguished (although this is not required in all embodiments).

[0209] Instantiating 1932 the free space criteria in the processor further includes specifying 1932B a left-right displacement range for the extended index finger. For example, the extended index finger must move at least a minimum distance from left to right without moving more than a maximum distance from left to right. Also, as described above, the input (in this case, a swipe gesture) in the example of FIG. 19 can be provided using the index finger even when enabled by the representation of an extended pinky finger shape.

[0210] Instantiating 1932 free space criteria in the processor also includes specifying 1932B a left-right displacement range for the extended index finger. For example, the extended index finger must move at least a minimum distance from left to right without moving more than a maximum distance from left to right. Also, as noted above, in the example of FIG. 19 , input (in this case, a swipe gesture) can be provided using the index finger even when enabled by the representation of the extended pinky finger shape.

[0211] Instantiating 1932 the free space reference in the processor further includes defining 1932C a sequence of downward, left, right, and upward motions. As discussed above with reference to other examples, these motions may be defined in terms of displacement, velocity, direction, envelope or range of position over a period of time, although other configurations may be equally suitable.

[0212] A free space input is sensed using a sensor (1934). The free space input is reported to a processor (1936). The processor determines (1938) whether the free space input meets a criterion. If the determination 1938 is positive, the method proceeds to step 1940; if the determination 1938 is negative, step 1940 is skipped. Continuing in step 1940, a surface-limited input response is invoked.

[0213] With reference collectively to Figures 14, 15A, 15B, 18, and 19, four approaches are illustrated, although embodiments are not limited to these approaches. For example, while the discrete / continuous distinction is detailed only with respect to the implicit approach shown in Figures 14, 15A, and 15B, the concepts of the discrete and / or continuous approaches may also be applied to explicit approaches, such as those shown in Figures 18 and 19. Any combination of passive / active, explicit / implicit, and / or continuous / discrete configurations, as well as other configurations, may be suitable for use in various embodiments.

[0214] Thus far, attention has been focused on identifying free-space gestures when defining free-space inputs and / or criteria, for example, to conveniently adapt existing gestures for touchscreens for use as free-space gesture inputs. As discussed herein above, enabling such adaptation through the identification of free-space gestures is a feature of various embodiments.

[0215] However, as noted above, embodiments are not limited to that feature. Another feature, but not necessarily limited to, can include signaling gestures and / or other free-space inputs so that the free-space inputs are available on systems configured for touchscreen or other surface-limited inputs. In other words, it shows how a free-space gesture can invoke a response if the touchscreen response is invoked on a system already configured for touchscreen input. This feature is described in more detail below.

[0216] FIG. 20 illustrates another exemplary method for applying free-space input to surface-limited control, utilizing an approach that invokes a response by providing a virtual or "pseudo" input.

[0217] In the example configuration of FIG. 20 , a data entity is instantiated in a processor (2030). The data entity is configured to invoke a reaction to input provided thereto. For example, in the case of a portable electronic device, the data entity can be an operating system. For such a device, the data entity receives input from a surface-limited input system and performs some reaction in response thereto. As a more specific example, a smartphone or head-mounted display can instantiate an operating system, such as Android, that is configured to accept touchscreen input and perform a reaction when the touchscreen input is provided to the device.

[0218] However, neither a touchscreen nor another surface-only input system is required. The ability to recognize touches from a touchscreen can be present in a data entity, but an actual touchscreen may or may not be present. Indeed, one beneficial feature of various embodiments (though not required) is the ability to utilize the ability to recognize surface-only input while providing free-space input.

[0219] Embodiments are not limited with respect to the nature of the data entity. Typically, although not necessarily, a data entity may be a collection of data and / or executable instructions instantiated in a processor, such as a computer program. In several places herein, an operating system is presented as an example of a data entity, although other configurations may be equally suitable.

[0220] Also, step 2030 of configuring the data entities is optional in at least certain embodiments and may or may not be considered part of the method. For completeness, Figure 20 shows configuring the data entities (2030), but typically the appropriate data entities may already be instantiated in the processor of the portable electronic device, such as the above example of an operating system. More colloquially, the operating system (or other data entities) may be "pre-existing" and not need to be configured.

[0221] As described herein above, an advantage of various embodiments is that existing infrastructure (devices, operating systems, software, etc.) already configured to accept touchscreens and / or other surface-only inputs can be adapted for free-space input, such as gestures. By enabling the use of existing infrastructure, new devices, systems, etc. can be backward compatible, eliminating the need to create new devices, operating systems, software, etc. In a more specific example, a head-mounted display can be controlled using free-space gestures while utilizing a smartphone processor, an Android or another mobile device operating system, and executing programs written to utilize the smartphone processor and / or an Android or another mobile device operating system, and utilizing similarly written libraries. In some cases, various embodiments can be implemented in addition to hardware, software, etc. that is not itself adapted for free-space gesture input.

[0222] 20, a free space input criteria is set in the processor 2032. The free space input is sensed using a sensor 2034.

[0223] The free space input is communicated to the processor 2036. Note that while the data entities are configured 2030 into the processor, the free space input is not necessarily communicated to the data entities (and typically is not, although other configurations may be suitable). It will be appreciated that various data entities, executable instructions, programs, etc. may be located on a single processor without necessarily communicating with each other.

[0224] The processor determines whether the free space input meets the criteria 2038. If the determination 2038 is positive, the method proceeds to step 2040; if the determination 2038 is negative, step 2040 is skipped.

[0225] Subsequently, in step 2040, a surface-bound input response is invoked, i.e., a response that typically follows a surface-bound input and occurs because the free-space input criteria are met. As a more specific example, an action that is a response to a touchscreen input is generated during a response to a free-space input.

[0226] Embodiments are not limited to how surface-limited input can be invoked (2040). The examples herein above do not address this issue in detail, and various approaches may be equally appropriate. However, it may be enlightening to present some example approaches to invoking surface-limited input responses, as shown in more detail in Figures 21, 22, and 23.

[0227] For example, as shown in the example of Figure 20, a surface-limited input reaction can be invoked within and / or by addressing a data entity (2040). As noted above, the data entity can be, for example, an operating system. An operating system for a portable electronic device with a touchscreen can include, for example, various reactions corresponding to various touchscreen inputs, along with executable instructions, data, and / or other "mechanisms" configured to recognize the touchscreen inputs and perform appropriate reactions thereto.

[0228] In the above configuration, as shown in Figure 20, step 2040 of invoking a surface-limited input response on a data entity may include establishing (2040A) a virtual surface-limited input link from the processor to the data entity. That is, some configuration may communicate with the data entity, and the data entity may recognize the communication as a surface-limited input, even though the communication is a virtual surface-limited input. More colloquially, a virtual or "pseudo" connection corresponding to the input system and / or the virtual input system itself is made when communicating with the data entity.

[0229] Note that, strictly speaking, the processor-to-data entity link 2040A can be considered a processor-to-processor link because the data entity already exists in the processor (established in step 2030). The link does not necessarily represent a physical connection. Rather, the link can take the form of a communications protocol, a data format, the activation of a port or communication path that already exists with the data entity, and the like. Various embodiments do not necessarily require the creation of a new physical connection (although such creation is not prohibited). Rather, the link can be established along an existing physical communication path in the processor (2040A) using existing executable instructions in the data entity (or other data entities).

[0230] 20, a virtual surface-bound input is generated in the processor based on the free-space input (2040B). That is, in the example of FIG. 20, a virtual input is generated that mimics an actual surface-bound input. In a more specific example, a virtual right swipe gesture is generated in the processor that is at least somewhat similar to an input that may be generated and / or provided from an actual touchscreen.

[0231] A virtual surface-limited input is generated from the free space input (2040B), and at least the generated virtual surface-limited input corresponds to a reaction that can be invoked with the free space input defined by the free space input criteria, i.e., if the free space input criteria (regardless of any details) define a right swipe, then the virtual surface-limited input is generated to be accepted by the data entity as a right swipe, and the data entity performs a reaction corresponding to the right swipe input via the surface-limited input system.

[0232] Typically, but not necessarily, virtual surface-bound input can have a form and / or content at least somewhat similar to real surface-bound input. For example, a real touchscreen can generate touchscreen data having a series of x and y coordinates representing points of contact with the touchscreen by one or more fingers over a period of time. A real touchscreen can also generate other data, such as applied pressure. Thus, a configuration of x and y coordinates and pressure over a period of time generated from a real touchscreen can represent a right swipe or other gesture input. Similarly, virtual touchscreen input can be generated to include x and y coordinates over a period of time similar to real touchscreen data for touchdowns and swipes (although in the case of virtual touchscreen input, the touchdowns and swipes do not actually occur).

[0233] In other words, the virtual x and y coordinates and pressure values ​​over a period of time can be at least approximately matched with the virtual x and y coordinates and pressure values ​​over a period of time generated by a real touchscreen during touchdowns and swipes. An exact match is not required, and the degree of match need not necessarily be high. As long as the virtual touchscreen input is accepted, the correspondence is sufficient. In some cases, a low degree of match is sufficient for accepted virtual touchscreen input (i.e., the virtual data is coarsely formatted, lacking content, or does not closely resemble the "real thing"), a low degree of match may be sufficient.

[0234] However, in certain embodiments, the virtual surface-limited inputs can be generated with some degree of precision and / or variability with the free-space inputs (2040B). For example, a broad or fast free-space swipe can generate a broad or fast virtual surface-limited swipe (2040B). With the above configurations, each virtual surface-limited input can be unique and / or matched to the details of the free-space input that generates the virtual surface-limited input (2040B).

[0235] However, in other embodiments, the virtual surface-bound inputs can be generated (2040B) to be predetermined and / or fixed. That is, any free-space swipe gesture (step 2038) that meets the criteria will result in a single reference virtual surface-bound swipe gesture, regardless of the details of the free-space swipe gesture. In such a case, a fast free-space swipe will generate the same surface-bound swipe as a slow free-space swipe. Colloquially, the virtual surface-bound inputs can be considered to be "pre-canned," or identical, for all swipe inputs.

[0236] Other options, including but not limited to hybrid approaches, may be equally appropriate. For example, virtual surface-bound input can be generated from free-space input in various ways, subject to constraints (2040B). For example, movements can be compressed or truncated, so that if a swipe gesture is wider than physically possible, such as on a smartphone touchscreen, the magnitude of the virtual swipe input can be reduced to avoid "error" responses.

[0237] Moreover, in at least certain embodiments, generating the virtual surface-bound input from the free-space input (2040B) may be sufficient to obviate certain other steps. For example, if the generated virtual surface-bound input is sufficiently realistic and / or “compelling” to the data entity, then the above-described determination 2038 may be omitted. That is, if the virtual touchscreen gesture is generated accurately enough (2040B), the data entity itself may be able to fully address the question of whether the input was made, and the virtual touchscreen input may be evaluated by the data entity without necessarily having to evaluate the actual free-space input (e.g., using existing means for evaluating actual touchscreen input).

[0238] Similarly, if the determination 2038 can be omitted, then instantiating the free space swipe criteria in the processor (2032) can also be omitted. If the determination is not made against the criteria, then the criteria themselves may not be necessary.

[0239] Continuing to refer to FIG. 20, the virtual surface limited input is communicated to the data entity by a virtual surface limited input link (2040C).

[0240] Thus, a response to the surface-limited input is invoked (2040). For at least some embodiments, it can be considered that at least some portion of performing the response occurs in the data entity. For example, if the data entity is an operating system for a portable electronic device, the operating system already has "mechanisms" in place to perform responses such as right swiping, tapping, pinching, etc. (as described above). Note that while various embodiments may communicate with the data entity to invoke functionality of the data entity to be performed in this manner, the data entity (e.g., an operating system, program, other data entity, etc.) is not necessarily part of various embodiments (although configurations in which the data entity is part of an embodiment are not excluded).

[0241] It should be noted that various embodiments may be provided to control systems, hardware, etc. when a response is invoked. In some embodiments, for example, a touchscreen smartphone running an operating system and program that does not have existing functionality to sense and / or respond to free-space gestures may be controlled by free-space gestures.

[0242] Next, FIG. 21 illustrates another exemplary method for applying free-space input to surface-limited control, again utilizing an approach that invokes a response by providing a virtual or "pseudo" input. The configuration of FIG. 21 is at least somewhat similar to the configuration illustrated and described above with reference to FIG. 20. However, for clarity, the configuration of FIG. 21 is illustrated more specifically with reference to a portable electronic device, particularly a head-mounted display (HMD) running a mobile operating system (OS), e.g., Android, iOS, Windows Phone, etc.

[0243] In the example configuration of Figure 21, a mobile operating system is instantiated 2130 on a processor of the head-mounted display. The mobile operating system is configured to invoke a reaction to provided touchscreen input. An actual touchscreen system may or may not be present in the head-mounted display. A touchscreen is acceptable but not required, as long as the mobile operating system accepts and / or recognizes touchscreen input.

[0244] As discussed above with reference to FIG. 20, step 2130 of instantiating a mobile operating system on a processor of the head mounted display is optional in at least certain embodiments and may or may not be considered part of the method.

[0245] 21, an interpreter is instantiated in the processor of the head mounted display 2132. The interpreter includes free space swipe criteria, i.e., criteria for determining whether a free space swipe gesture has been made.

[0246] The term "interpreter" refers to a possible, but not required, approach to performing certain steps. In certain cases, it may be advantageous to incorporate one or more free-space input criteria, executable instructions for determining whether an input meets the criteria, and communicating with an operating system into a substantially integrated assembly. More colloquially, some or all of the functionality of various embodiments may be incorporated into a program that can be conveniently loaded into the processor of a head-mounted display or other portable electronic device so that the device can be controlled through free-space gestures. Such a program can be thought of as interpreting new "input languages" (free-space gestures) and fully or partially implementing the behavior of existing input languages ​​(e.g., a mobile operating system with touchscreen input capabilities). Therefore, such programs are referred to herein as interpreters.

[0247] For illustrative purposes, the method shown in Figure 21 includes the interpreter described above (certain following figures may also refer to the interpreter concept), however, both the use of an interpreter and the illustrated configuration for the interpreter are merely examples, and other configurations may be equally suitable.

[0248] 21, the free-space swipe is sensed using a sensor in the head-mounted display 2134. Typically, but not necessarily, the sensor may be a camera, depth camera, or similar camera, although other configurations may be equally suitable.

[0249] The free space swipe input is reported from the sensor to the interpreter 2136. The interpreter determines whether the free space swipe meets a criteria 2138. If the determination 2138 is positive, the method proceeds to step 2140; if the determination 2138 is negative, step 2140 is skipped.

[0250] Continuing at step 2140, a touchscreen swipe response is invoked on the mobile operating system. In the configuration shown in FIG. 21, a virtual touchscreen link is connected from the interpreter to the mobile operating system (2140A). Typically, but not necessarily, the mobile operating system may include a protocol for identifying and communicating with a physical touchscreen. In connecting the virtual touchscreen link from the interpreter to the mobile operating system, the mobile operating system's protocol may cause the mobile operating system to recognize the interpreter (or a portion thereof) as a physical touchscreen and to recognize at least some information sent from the interpreter as touchscreen input from a physical touchscreen. More colloquially, the interpreter tells the mobile operating system, "I'm a touchscreen," and arranges for touchscreen data to be sent to the mobile operating system. However, the interpreter typically does not represent an actual touchscreen, but rather a virtual version of it (and corresponding virtual data).

[0251] In other words, in a sense, the interpreter can be thought of as "fooling" the mobile operating system by connecting a pseudo (i.e., virtual) touchscreen to the mobile operating system, which senses the pseudo (i.e., virtual) touchscreen input.

[0252]

[00130] Still referring to Figure 21, virtual surface touchscreen swipes are generated in the interpreter based on the free-space swipes (2140B). That is, virtual inputs are generated that mimic actual swipes from an actual touchscreen (2140B). As discussed above with reference to Figure 20, virtual touchscreen swipes are generated (2140B) to be unique for each free-space swipe, and can be scaled across all free-space swipes, hybrids thereof, etc.

[0253] The virtual touchscreen swipe is communicated to the mobile operating system by the virtual touchscreen link (2140C). In Figure 21, the method ends after 2140C (at which point the touchscreen swipe response has already been invoked within the mobile OS (2140)), but actions performed by the mobile operating system, head-mounted display, etc. in performing the associated response may continue for an indefinite period of time. For example, as shown in Figure 21, the method is complete once the touchscreen swipe response is invoked (2140), but the invoked action, such as controlling the head-mounted display in some way, e.g., by moving display content from left to right, invoking a menu, or other function (which may, but need not, include physical responses to the head-mounted display and / or other hardware and / or communicating systems), may be performed and / or continue after the steps shown in Figure 21 are nominally complete.

[0254] With particular reference to step 2140B, it should be noted that certain operating systems, programs, and / or other data entities may utilize an approach referred to as “syntactic sugar.” Syntax sugar represents a configuration in which certain inputs and / or other data may be defined along with criteria, and / or such characteristics may be standardized to recognize, notify, and respond to inputs. As a more specific example, in a mobile operating system configured to receive touchscreen input, the requirements for an input to be accepted as a right swipe, the processing of inputs that are or are suspected to be right swipes, and the commands or other responses to be performed upon receiving a right swipe input may be embodied as a collection of data and / or executable instructions. More colloquially, “building blocks” of code (also referred to as syntactic sugar as described above) also exist for right swipes, and when creating a program to respond to a right swipe, the building blocks of code may simply be added. The use of syntactic sugar may be considered a work-saving approach, eliminating the need to define a swipe and write code to detect and identify the swipe for each individual program. Program coding standards can also be simplified.

[0255] Using syntactic sugar in this manner can, at least in some embodiments, simplify execution.

[0256] For example, consider mobile operating systems for electronic devices such as Android, iOS, Windows Phone, etc. The operating system may function, in part, to mitigate and support the use of various programs, libraries, etc. running on a processor within the portable electronic device. The range of programs and other software that the operating system can accommodate is likely quite broad, and similarly, a given operating system may be used on a variety of different electronic devices. For simplicity's sake, it may be beneficial to standardize at least certain features, such as (among other things) touchscreen gesture input, so that gesture input is consistent across various programs and / or devices. More colloquially, it may be beneficial for "a right swipe is a right swipe" with respect to at least some, if not all, programs, devices, etc.

[0257] Syntactic sugar can help accomplish this scaling. For example, if a touchscreen right swipe block is included and / or made available in an operating system, using a right swipe as a touchscreen input can simply utilize the block rather than coding input criteria, decision making, command invocation, etc. from the start.

[0258] If syntactic sugar and / or other standardized approaches exist, they can be utilized in at least certain embodiments. Continuing with the example above, if a consistent touchscreen right swipe is implemented in some or all electronic devices running an operating system and / or some or all programs running on the electronic devices, it would be beneficial to configure executable instructions (and / or other approaches) that generate a virtual touchscreen swipe (2140B) based on that consistent implementation. If a wide range of devices and / or software handle touchscreen inputs in the same way, even to the extent that they use the same or similar code (i.e., syntactic sugar), the process of invoking touchscreen inputs can be simplified.

[0259] Thus, syntactic sugar can represent an opportunity to simplify practical and / or efficient execution. A set of executable instructions configured to generate a virtual touchscreen swipe (2140B) can be written to correspond to the syntactic sugar for a touchscreen right swipe, thereby providing a virtual touchscreen swipe that can be reliably predicted to be accepted as an actual touchscreen swipe by various operating systems, programs, electronic devices, etc., and thus a corresponding response can be reliably executed.

[0260] Note that syntactic sugar is not necessarily part of various embodiments, and neither syntactic sugar nor other approaches to scaling input and / or other functionality of an operating system and / or electronic device are necessarily performed by, motivated by, or required by various embodiments. Rather, such scaling may already exist in at least a particular operating system, program, electronic device, etc., and various embodiments may utilize that scaling to perform particular functions.

[0261] However, even if syntactic sugar is not used by the operating system, electronic device, software, etc., embodiments may be utilized with the operating system, electronic device, software, etc. In the absence of syntactic sugar and / or other standardization, a more precise set of executable instructions (or other approach) may be utilized to generate (2140B) a virtual touchscreen swipe that is acceptable as an actual touchscreen swipe, e.g., to provide a virtual touchscreen swipe that is realistic enough to meet the standards of various programs, electronic devices, etc.

[0262] Considering syntactic sugar in certain embodiments is in some ways similar to considering a standard communication protocol between an actual touchscreen and a processor within a touchscreen device. Such a standard communication protocol may exist, for example, to facilitate compatibility between various touchscreens and various devices. Such touchscreen communication standards may not be created or included in various embodiments, may not be part of various embodiments, and need not be present in all embodiments. However, various embodiments may utilize such communication standards, for example, when connecting virtual touchscreen links (2140A) and / or announcing virtual touchscreen swipes (2140C).

[0263] Similarly, syntactic sugar may not be created or included in various embodiments, may not be part of various embodiments, and need not be present in all embodiments. However, various embodiments may utilize syntactic sugar when generating 2140B virtual touchscreen swipes (and similarly other virtual inputs). While the use of syntactic sugar is mentioned herein as a specific example of how certain functions might actually be performed, embodiments are not limited thereto.

[0264] Referring to FIG. 21 and step 2140, the exemplary configuration effectively invokes 2140 a touchscreen swipe response by providing simulated touchscreen input to a mobile operating system. Such a configuration is beneficial, at least for certain embodiments, because, for example, a mobile operating system configured for use with a touchscreen device is already capable of accepting touchscreen input. That is, protocols for configuring touchscreen input may exist in the mobile operating system, as may channels for reporting touchscreen input, allowing the mobile operating system to "predict" such touchscreen input (so that antivirus software, error trapping routines, and the like are less likely to interpret the virtual touchscreen input as spoofed data, an attack, or the like). Colloquially, a touchscreen device may be configured to receive touchscreen input, thereby simplifying the transmission of simulated (virtual) touchscreen input.

[0265] However, embodiments are not limited to using virtual touchscreen input to invoke responses, and other configurations may be equally suitable.

[0266] Figure 22 shows another example method of applying free-space input to surface-bound control. However, while the configuration of Figure 21 uses virtual touchscreen input to invoke a response, the example of Figure 22 uses virtual touchscreen events to invoke a response.

[0267] In the example configuration of Figure 22, a mobile operating system is instantiated on a processor of the head-mounted display 2230. The mobile operating system is configured to invoke a reaction to provided touchscreen input.

[0268] An interpreter is instantiated in the processor of the head mounted display 2232. The interpreter includes free space swipe criteria, i.e., criteria for determining whether a free space swipe gesture has been provided.

[0269] A free-space swipe is sensed using a sensor in the head-mounted display (2234). The free-space swipe input is reported from the sensor to an interpreter (2236). In the interpreter, it is determined (2238) whether the free-space swipe meets a criterion. If decision 2238 is positive, the method continues at step 2240; if decision 2238 is negative, step 2240 is skipped.

[0270] Subsequently, in step 2240, a touchscreen swipe response is invoked in the mobile operating system.

[0271] With respect to the term "event" as used in 2240, a mobile operating system may typically, although not necessarily, include internal configurations that track events, such as events associated with touchscreen input. When touchscreen input is received by a mobile operating system and determined to represent (for example) a swipe, the touchscreen input itself (detailed information about where and when the touch occurred, the contact path during the swipe, the speed of movement, etc.) cannot be forwarded within the operating system. Rather, an event occurs within the operating system and can then be forwarded. For example, when touchscreen swipe input is received from the touchscreen in the operating system, a flag may be set and some form of logical state, such as "swipe=true," may be set as an indication that swipe input was received.

[0272] In the configuration shown in Figure 22, a virtual event link is connected from the interpreter to the mobile operating system (2240A), which may represent, for example, identifying the part of the operating system where the event is recorded (if any), executable instructions within the operating system that control whether the event is determined to have occurred, etc.

[0273] 22, virtual touchscreen swipe events are generated in the interpreter corresponding to events that would occur within the operating system if an actual virtual touchscreen swipe were provided and accepted by the operating system (2240B). That is, virtual events are generated in the interpreter that mimic actual events that occur within the mobile operating system (2240B).

[0274] The virtual touchscreen swipe event is notified to the mobile operating system by the virtual event link (2240C). At this point, as far as the mobile operating system is concerned, the touchscreen swipe has already occurred and a corresponding reaction can be performed by the mobile operating system.

[0275] In a sense, the approach shown in FIG. 22 can be thought of as a "workaround" that bypasses the touchscreen system. Touchscreen input data, whether real or virtual, may or may not be sent to the operating system. Receipt and / or consideration of touchscreen data, whether real or virtual, is irrelevant. Rather, the portion of the operating system that "knows" or "determines" whether a touchscreen swipe occurred will cause the touchscreen swipe to be accepted as having actually occurred. If touchscreen input is thought of as a record made by someone keeping score of a game, then the event is analogous to receiving a score. Where the configuration of FIG. 21 produces a pseudo-analogy of a recorded play, the configuration of FIG. 22 directly modifies the score.

[0276] An arrangement such as that shown in FIG. 22 may be effective for at least certain embodiments. For example, touchscreen inputs are relatively complex and can be evaluated by multiple, possibly sophisticated, criteria. Thus, it may be difficult to generate virtual touchscreen inputs sufficient to "fool" a mobile operating system. If the system is not standardized with syntactic sugar, for example, as described above, the specific criteria that the virtual touchscreen input must meet may vary from device to device, operating system to operating system, and / or program, again requiring more sophisticated impersonation to generate appropriate virtual touchscreen inputs.

[0277] In contrast, inputs are typically relatively simple data, such as "occurred" or "did not occur," and possibly even a single bit of data. Generating virtual events (2240B) can be relatively straightforward, at least for certain embodiments. Connecting virtual event links (2240A) and / or notifying virtual touchscreen swipe events (2240C) can be challenging. For example, while mobile operating systems designed to interface with touchscreens may be easily configured to accept touchscreen input and may provide clear paths for the input provided, such mobile operating systems may not be easily configured to accept events from external sources, where the events are typically generated within the mobile operating system.

[0278] It should be understood that suitable approaches for implementing various embodiments may vary depending on the details of a given embodiment: in some cases, a configuration such as that shown in Figure 21 may be suitable, in other cases, a configuration such as that shown in Figure 22 may be suitable, or other configurations may be suitable.

[0279] Next, Figure 23 shows another example method of applying free space input to surface-bound controls. The example in Figure 23 invokes a response by sending a command to the mobile operating system.

[0280] In the example configuration of Figure 23, a mobile operating system is instantiated (2330) on a processor of a head-mounted display. The mobile operating system is configured to invoke a response to touchscreen input provided to the system. An interpreter is instantiated (2332) on the processor of the head-mounted display. The interpreter includes free-space swipe criteria. A free-space swipe is sensed (2334) using a sensor on the head-mounted display. The free-space swipe input is communicated (2336) from the sensor to the interpreter. In the interpreter, it is determined (2338) whether the free-space swipe meets the criteria. If the determination 2338 is positive, the method proceeds to step 2340; if the determination 2338 is negative, step 2340 is skipped.

[0281] Continuing at step 2340, a touchscreen swipe response is invoked in the mobile operating system.

[0282] In the configuration shown in Figure 23, a virtual command link is connected from the interpreter to the mobile operating system (2340A). This may represent, for example, identifying the portion of the operating system that generates commands in response to touchscreen input, the portion of the operating system that communicates the commands to a destination, etc. The configuration of Figure 21 provides virtual touchscreen input, the configuration of Figure 22 inserts virtual events, and the configuration of Figure 23 sends virtual commands to the operating system.

[0283] Virtual operating system commands are generated in the interpreter (2340B) that correspond to commands that could be generated if actual virtual touchscreen swipes were provided to and accepted by the operating system. That is, virtual commands are generated in the interpreter (2340B) that mimic actual commands to the mobile operating system. Typically, although not necessarily, the virtual commands may be commands that cause the operating system to perform an invoked response and / or commands that instruct other entities (applications, video drivers, etc.) to perform an invoked response.

[0284] The virtual operating system command is communicated to the mobile operating system by the virtual event link 2340C. At this point, given the virtual command, a corresponding reaction can be performed by the mobile operating system.

[0285] While the approach shown in FIG. 22 cannot be considered a "workaround" that circumvents the touchscreen system, the approach shown in FIG. 23 can be considered a "workaround" that circumvents most or all of the input and the operating system's evaluation of the touchscreen input. Touchscreen input data and associated events, whether real or virtual, may or may not be transmitted. Rather, the configuration of FIG. 23 allows appropriate commands to be sent directly to the operating system without following the normal process used by the operating system in generating such commands. If the touchscreen input corresponds to recording a game score and the events correspond to points gained, the commands correspond to declaring victory (or defeat). The configuration of FIG. 23 directly instructs the operating system to perform some function via pseudo-commands sent to the operating system.

[0286] An arrangement such as that shown in FIG. 23 may be effective for at least certain embodiments. For example, sending commands may be considered a very straightforward approach in that no virtual data is generated and no virtual events are recorded. In fact, the functionality of the operating system itself may be performed, at least to some extent, by simply issuing commands rather than causing an action that determines that such commands should be issued. However, as discussed above with reference to FIG. 22, mobile operating systems are not necessarily configured to easily accept internal commands from external sources. In fact, the above functionality may be considered subversive of the operating system and may be circumvented through measures that protect the operating system (e.g., antivirus software).

[0287] 21-23 illustrate three example approaches that can be used to implement various embodiments, but the embodiments are not so limited, and other approaches may be equally suitable. For example, in certain embodiments, commands may be generated in an interpreter and communicated to an application, driver, or the like within a head-mounted display or other device, bypassing the mobile operating system entirely. Also, hybrid configurations that combine multiple approaches may be appropriate. The illustrated example approaches are not necessarily exclusive.

[0288] Furthermore, while the examples shown in Figures 21-23 refer for clarity to a single case, namely, a head-mounted display with an interpreter and a mobile operating system instantiated on a processor, embodiments are not limited to such a configuration, and other devices, systems, approaches, etc. may be equally suitable.

[0289] As noted above, examples have been presented herein relating to at least two distinct features: adapting free-space input as the equivalent of a surface-limited input, and invoking a response to a free-space input in a system configured to accept a surface-limited input. For simplicity, these two features have been described separately, although combinations of these features may be appropriate within various embodiments. An example is shown in Figures 24A and 24B. However, it should be emphasized that while these two features have been described in detail herein, embodiments are not limited to the illustrated features.

[0290] Referring to Figure 24A, a mobile operating system is instantiated on a processor of the head mounted display (2430). The mobile operating system is configured to invoke a reaction to the provided touchscreen input. An interpreter is instantiated on the processor of the head mounted display (2432). The interpreter includes free space swipe criteria.

[0291] With respect to instantiating 2432 the interpreter, and specifically instantiating the free-space input criteria, the example of Figure 24A presents a discrete implicit approach (at least somewhat similar to the approach illustrated and described with reference to Figure 15A). Three sub-criteria for touch-down, swipe, and lift-off states are instantiated in the processor in steps 2432A, 2432B, and 2432C, respectively.

[0292] Referring to subsystem 2432A, instantiating the free-space touchdown condition subcriteria in the processor includes requiring the extended index finger to act as the end effector to provide free-space input (2432A1) and excluding extension of other fingers (2432A2). The free-space touchdown condition subcriteria prescribe a range of downward motion (2432A3), limit rebound acceleration (2432A4), and also limit the range of motion of the finger joints (2432A5).

[0293] 24A (and step 2432) and subsystem 2432B, instantiating the free-space touchdown condition subcriteria in the processor includes requiring the extended index finger to act as the end effector to provide free-space input (2432B1) and excluding extension of other fingers (2432B2). The free-space touchdown condition subcriteria also define a range of left-right motion (2432B3), limit rebound acceleration (2432B4), and limit the range of motion of the finger joints (2432B5).

[0294] For subsystem 2432C, instantiating the free-space touchdown condition subcriteria in the processor includes requiring the extended index finger to act as the end effector to provide free-space input (2432C1) and excluding extension of other fingers (2432C2). The free-space touchdown condition subcriteria prescribe a range of upward motion (2432C3), limit rebound acceleration (2432C4), and also limit the range of motion of the finger joints (2432C5).

[0295] Next, referring to Figure 24B, a free-space swipe is sensed using a sensor in the head-mounted display (2434). The free-space swipe input is communicated from the sensor to an interpreter (2436). In the interpreter, it is determined (2438) whether the free-space swipe meets a criterion. If the determination at 2438 is positive, the method proceeds to step 2440; if the determination at 2438 is negative, step 2440 is skipped.

[0296] Continuing at step 2440, a touchscreen swipe response is invoked in the mobile operating system. A virtual touchscreen link is connected from the interpreter to the mobile operating system (2440A). A virtual surface touchscreen swipe is generated in the interpreter based on the free space swipe (2440B). The virtual touchscreen swipe is communicated to the mobile operating system by the virtual touchscreen link (2140C).

[0297] FIG. 25 is a schematic diagram illustrating an exemplary embodiment of the device.

[0298] In the example configuration of Figure 25, device 2560 includes a processor 2562 and a sensor 2564 in communication with the processor. The processor further includes an interpreter 2568. The configuration of Figure 25 also shows a data entity 2566 located on processor 2560. As noted above, a data entity (such as an operating system) need not necessarily be part of various embodiments, but may be present and in communication with various embodiments. To emphasize this point, data entity 2566 is shown with a dashed line.

[0299] The processor 2562 is configured to execute executable instructions that are instantiated therein. The present invention is not limited with respect to the selection of the processor 2562. Suitable processors 2562 include, but are not limited to, digital electronic microprocessors. For clarity, the processor 2562 is referred to at least in some places herein as a separate physical device, although this is not required and other configurations may be suitable. For example, the processor 2562 may comprise two or more physical processors working together, processing functions in a network that do not have a distinct physical form, etc.

[0300] Sensor 2564 is configured to sense free-space input. As shown in FIG. 25, sensor 2564 is an imaging device such as a color digital camera, a depth camera, or the like. However, these are merely examples, and embodiments are not limited with respect to what sensors may be utilized. Suitable sensors may include, but are not limited to, digital and / or analog cameras (whether operating in achromatic and / or color, or sensing visible light, infrared light, thermal radiation, and / or ultraviolet light), depth cameras, structured light sensors, time-of-flight sensors, and ultrasonic sensors.

[0301] As shown, the sensor 2564 is directly connected to the processor 2562. However, this is merely one example. Other configurations may be equally suitable, including, but not limited to, configurations in which the sensor 2564 is not part of the device 2560 and / or in which the sensor 2564 is physically separate from the device 2560. Additionally, embodiments are not limited with respect to how the processor 2562 and the sensor 2564 may communicate with each other. While a hardwired link is shown in FIG. 25 as being suitable, other configurations may be equally suitable, including, but not limited to, wireless communication.

[0302] Data entity 2566, as noted above, can be an optional feature, at least for certain embodiments, and if present, may or may not be part of device 2560 itself (as indicated by the dashed line). If present, data entity 2566 can be configured to perform, or at least direct the performance of, some function, such as a surface-limited input reaction, when invoked by various embodiments. Embodiments are not limited with respect to what, if any, data entity 2566 may be present and / or whether data entity 2566 is present at all. Also, while the configuration of FIG. 25 shows data entity 2566 located on processor 2562 and interpreter 2568 located on the same processor 2562, this is merely an example.

[0303] If present, data entity 2566 may reside in another processor. For example, apparatus 2560 may have processor 2562 with interpreter 2568 that communicates with another processor. In a more specific example, apparatus 2560 may exist as a "plug-in" device configured to be connectable to another device to provide additional functionality. In the example above, the plug-in device may have processor 2560 separate from any processor that may reside in the other device. Similarly, sensor 2564 illustrated as part of apparatus 2560 may be a sensor on another device with the functionality of sensor 2564 being utilized for purposes of various embodiments.

[0304] 25, as noted above, interpreter 2568 is located in processor 2562. As noted above, interpreter 2568 may include and / or represent embodiments of two or more functions, such as a free space input criterion and a comparison means to determine whether the free space input satisfies the criterion. However, the use of interpreter 2568 is merely exemplary, and other configurations may be equally suitable, including, but not limited to, configurations in which the free space input criterion and comparison means are separately located in processor 2562.

[0305] FIG. 26 is a schematic diagram illustrating another exemplary embodiment of the device.

[0306] In the example configuration of Figure 26, device 2660 includes a processor 2662 at least similar to the processor described with reference to Figure 25. Device 2660 further includes sensors 2664A and 2664B in communication with the device. Sensors 2664A and 2664B may be sensors at least similar to the sensors described with reference to Figure 25. However, as shown, embodiments are not limited to a single sensor. Similarly, other components may be duplicated, combined, or separated.

[0307] 26 further includes a communication means 2682 in communication with the processor 2662 and configured to communicate between the processor and other entities. When present, the communication means 2682, if present, is not limited with respect to the communication means 2682. Typically, although not necessarily, the communication means 2682 may be a wireless communication means such as Wi-Fi or Bluetooth, although other configurations may be equally suitable.

[0308] The device 2660 includes a display 2684 in communication with the processor 2662. As noted elsewhere herein, various embodiments may be applied to portable electronic devices, including, but not limited to, head-mounted displays. An embodiment does not necessarily require a display 2684; the display 2684 present in certain embodiments herein is an example. As shown in FIG. 26 , the display 2684 is a stereo display having left and right screens adapted for output to the viewer's left and right eyes. However, this is merely an example. The embodiments are not particularly limited with respect to the type of display 2684. Typically, but not necessarily, the display 2684 can be a visual display. When present, various devices are suitable for use as the display 2684, including, but not limited to, light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), plasma screen panels (PDPs), liquid crystal displays (LCDs), and the like. Similarly, the use of a projection or transmission display, in which the viewing surface is a receiving surface for images generated elsewhere and then projected or otherwise transmitted, may also be appropriate. Other configurations may also be equally suitable, including, but not limited to, systems that display images directly to the user's eyes. Either digital or analog display technologies may be suitable. Furthermore, embodiments are not limited to using a visual display as display 2684.

[0309] With further reference to FIG. 26 , the illustrated apparatus 2660 includes a data storage device 2686, such as a hard drive or solid-state drive, in communication with the processor 2662. Neither the data storage device 2686 nor its functionality is required in all embodiments. However, the above examples and comments herein regarding methods and apparatus also include reference to data storage devices and / or other devices / systems as facilitating instantiation of free-space input criteria into the processor. In a more specific example, the data storage device 2686 shown in FIG. 26 can store information related to the above-described configuration, which information is accessed by and / or instantiated into the processor 2662. Embodiments are not limited with respect to including and / or connecting to components such as the data storage device 2686. While not necessarily required, components such as the data storage device 2686 (or communication means, display, etc.) are not prohibited. Also, embodiments are not limited with respect to what data storage device 2686 is suitable. Although data storage device 2686 is shown in FIG. 26 as a hard drive, this is merely an example and other configurations may be equally suitable.

[0310] The processor 2662 includes an instantiated interpreter 2668. Typically, although not necessarily, the interpreter 2668 and / or its components can include executable instructions, data, and the like.

[0311] As shown, interpreter 2668 includes free space input reference 2670, which includes state references: state A reference 2670A, state B reference 2670B, state C reference 2670C, and state D reference 2670D. ​​While the use of state references 2670A-2670D has been discussed previously herein, this is merely an example. Other configurations, including but not limited to those detailed herein, may be equally suitable.

[0312] Interpreter 2668 includes boundary criteria 2672. Boundary criteria have been described previously herein. Note that the above examples do not necessarily combine state criteria (e.g., 2670A-2670D) with boundary criteria 2672. Similarly, not all embodiments that include state criteria 2670A-2670D necessarily include boundary criteria 2672, and vice versa. Both criteria are shown together in this specification as examples (embodiments that include both are not excluded).

[0313] The interpreter 2668 includes an input comparison means 2674 configured to compare the free space input (eg, received from sensors 2664A and / or 2664B) with the free space input reference 2670 and / or state references 2670A-2670D.

[0314] The interpreter 2668 further includes an invoking means 2676 configured to invoke a touchscreen input response. As shown in Figure 26, the invoking means 2676 includes a virtual touchscreen link 2678A and a virtual input generator 2680A, a virtual touch input event link 2678B and a virtual event generator 2680B, and a virtual operating system command link 2678C and a virtual command generator 2680C. As mentioned above, each of the above pairs can be configured to invoke a touchscreen input response in different ways. Typically, although not required, a given embodiment may include only one of the above pairs: virtual touchscreen link 2678A and virtual input generator 2680A (configured to signal and generate virtual touchscreen inputs, respectively), or virtual touch input event link 2678B and virtual event generator 2680B (configured to signal and generate virtual events, respectively), or virtual operating system command link 2678C and virtual command generator 2680C (configured to signal and generate virtual commands, respectively). Although all three pairs are shown in Figure 26 for completeness, typically only one pair need be present (although embodiments having more than one pair are not excluded).

[0315] As discussed above with reference to FIG. 25, although certain components of device 2660 are shown to be and / or be part of interpreter 2668, similar components and / or similar functionality may be incorporated into various embodiments of the device without necessarily utilizing interpreter 2668.

[0316] The device 2660 may include and / or may cope with an operating system 2666 instantiated on a processor 2662, including but not limited to a mobile operating system.

[0317] Next, referring to Figure 27, embodiments are not particularly limited in terms of shape. Suitable configurations include, but are not limited to, the example shown in Figure 27, which illustrates a device configured in the shape of a head-mounted display similar to glasses.

[0318] As shown in Figure 27, an exemplary embodiment of device 2760 includes a body 2788 having a shape similar to eyeglasses and configured to be worn like eyeglasses. A processor 2762 configured to execute executable instructions is disposed in body 2788. Although not visible as a separate entity, processor 2762 may support an interpreter and data entity as shown in Figure 25, an interpreter (including free space input criteria and / or state criteria, boundary criteria, input comparison means, calling means including virtual touchscreen links and virtual input generation means, virtual input event links and virtual event generation means, or virtual operating system command links and virtual command generation means) and operating system as shown in Figure 26, and / or other configurations of executable instructions and / or data.

[0319] Device 2760 of Figure 27 includes communication means 2782 and data storage device 2786, both located on body 2788. Device 2760 further includes sensors 2764A and 2764B located on body 2788 and shown in Figure 27 as imaging devices in a stereo configuration, although these are merely examples. Device 2760 further includes displays 2784A and 2784B located on body 2788 and shown as left and right visual displays in a stereo configuration.

[0320] It should be noted that in the illustrated configuration, when body 2788 is configured, sensors 2764A and 2764B are disposed on the body, and a viewer wears body 2788, the fields of view of display motion sensors 2764A and 2764B are approximately aligned with the viewer's line of sight and may include a field of view at least somewhat comparable to the viewer's field of view, assuming that display motion sensors 2764A and 2764B exhibit a field of view somewhat similar to that of the viewer. Similarly, in the illustrated configuration, when body 2788 is configured, displays 2784A and 2784B are disposed on the body, and a viewer wears body 2788, displays 2784A and 2784B are proximate and generally in front of the viewer's eyes.

[0321] However, it is emphasized that the configuration of FIG. 27 is merely one example, and other configurations may be equally suitable, including, but not limited to, other head-mounted displays and non-head-mounted display devices and systems.

[0322] Figure 28 is a block diagram of an apparatus capable of performing various operations and storing various information generated and / or used by those operations, according to one embodiment of the disclosed technique. The apparatus may represent a computer or processing system as described herein. Processing system 2890 is a hardware device capable of implementing other entities, components, or services shown in the examples of Figures 1-27 (and any other components described herein). Processing system 2890 includes one or more processors 2891 and memory 2892 coupled to interconnect 2893. Interconnect 2893 is shown in Figure 28 to abstractly represent one or more separate physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers. Thus, the interconnect 2893 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electronics and Electrical Engineers (IEEE) Standard 1394 bus, also known as "Firewire."

[0323] Processor 2891 is the central processing unit of processing system 2890 and therefore controls the overall operation of processing system 2890. In particular embodiments, processor 2891 accomplishes this by executing software or firmware stored in memory 2892. Processor 2891 may be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), trusted platform modules (TPMs), etc., or combinations of the above devices.

[0324] Memory 2892 is or includes the main memory of processing system 2890. Memory 2892 represents any form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of these devices. When in use, memory 2892 may include code. In one embodiment, the code includes generic programming modules configured to recognize generic programs received via a computer bus interface and create generic programs for execution on the processor. In another embodiment, the generic programming modules may be implemented using hardware circuitry such as an ASIC, PLD, or field programmable gate array (FPGA).

[0325] Network adapter 2894, storage device 2895, and I / O devices 2896 are also connected to processor 2891 through interconnect 2893. Network adapter 2894 provides processing system 2890 with the ability to communicate with remote devices over a network and may be, for example, an Ethernet adapter or a Fibre Channel adapter. Network adapter 2894 may also provide processing system 2890 with the ability to communicate with other computers in a cluster. In some embodiments, processing system 2890 may use two or more network adapters, one to handle communications within and one to handle communications outside the cluster.

[0326] The I / O devices 2896 may include other input and / or output devices, such as, for example, a keyboard, a mouse or other pointing device, a disk drive, a printer, a scanner, a display device, etc. The I / O devices 2896 may further include, for example, a camera and / or other imaging device configured to accept visual input, including, but not limited to, poses and / or gestures. The display devices may include, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), or other applicable or convenient display device. The display devices may take a variety of forms, including, but not limited to, stereo displays suitable for use in near-eye applications such as head-mounted displays or other wearable devices.

[0327] Code stored in memory 2892 can be executed as software and / or firmware to program processor 2891 to perform the operations described herein. In particular embodiments, the software or firmware can be initially provided to processing system 2890 by downloading it from a remote system through processing system 2890 (e.g., via network adapter 2894).

[0328] The techniques described herein can be implemented, for example, by programmable circuitry (e.g., one or more microprocessors) programmed with software and / or firmware, by entirely hardwired (non-programmable) circuitry, or a combination of these forms. Hardwired circuitry can take the form of, for example, one or more AISCs, PLDs, FPGAs, etc.

[0329] The software or firmware used to perform the techniques presented herein can be stored on a machine-readable storage medium and executed by one or more general-purpose or special-purpose programmable microprocessors. As used herein, the phrase "machine-readable storage medium" includes any mechanism capable of storing information in a form accessible by a machine.

[0330] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, an iPhone®, a Blackberry, a processor, a telephone, a web appliance, a network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed by the machine.

[0331] The machine-accessible storage medium or storage device 2895 includes, for example, storage / non-storage media (e.g., ROM; RAM; magnetic disk storage media; optical storage media; flash memory devices; etc.), or the like, or a combination thereof. Typically, storage media are or can include non-transitory devices. In this context, non-transitory storage media can be tangible devices, i.e., devices that have a concrete physical form, but the devices can also change physical state. Thus, for example, non-transitory refers to devices that remain tangible despite a change in state.

[0332] The term "logic" as used herein may include, for example, programmable circuitry that is programmed with specific software and / or firmware, application specific hardwired circuitry, or a combination thereof.

[0333] The above description, examples and data provide a complete description of the manufacture and construction of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims appended hereto.

Claims

1. determining, by a processor, a free space input criterion for a free space input, the free space input being received within an augmented reality structure defined by a volume boundary; sensing the free space input within the volume boundary with a sensor; comparing, by the processor, the free space input to the free space input reference; determining, by the processor, that the free-space input includes movement of the index finger while the index finger and little finger are extended, thereby identifying the hand movement as an input gesture and excluding movement of the index finger while the little finger is not extended; determining, by the processor, that the free-space input satisfies the free-space input criteria; responsive to the free-space input satisfying the free-space input criteria, executing, by the processor, a surface-limited input response associated with the free-space input; A method comprising:

2. 10. An apparatus comprising a sensor and a processor, the sensor and processor configured to perform the method of claim 1.

3. A non-transitory computer-readable storage medium storing a program that causes a computer to execute a process, the process comprising the method of claim 1.

Images