Locating tap locations on handheld electronic devices based on inertial measurements

Abstract

Systems and methods are described that use a data stream from an integrated inertial measurement unit (IMU) to identify the location of taps on the body of a handheld device in real time. Taps are generated by tapping the handheld device with an object (e.g., a finger) or by moving the handheld device and tapping it against another object. Measurements of the IMU accelerometer, gyroscope, and / or orientation (relative to the Earth's magnetic field and / or gravity) are examined to find a signature that distinguishes taps at one location on the device body compared to signal characteristics generated by taps at other locations. Neural networks and / or numerical methods can be used to perform such classification. Tap location, tap timing, and tap attributes, such as the magnitude of the applied force, the device's orientation, and the amplitude and direction of movement during and after the tap, can be used to control or adjust responses within the handheld device and / or actions in connected devices.

Description

[Technical Field]

[0001] This application relates generally to systems and methods for individuals to perform machine-based interactions using handheld electronic devices. The handheld devices can be used by anyone, but are particularly suited for use by young children, utilizing simple interaction actions that do not require precise manual dexterity and / or understanding of complex interaction sequences. The systems and methods herein employ techniques from the fields of computer programming, electronic design, firmware design, inertial measurement units (IMUs), ergonomic construction, device control, human motion control, and human-machine interaction. The systems and methods can provide users, particularly young children, with an intuitive machine interface for quickly and / or instinctively interacting within an environment composed of real and / or virtual objects.

[0002] Related application data This application claims the benefit of and priority to U.S. Application No. 17 / 874,253, filed July 26, 2022, the entire disclosure of which is expressly incorporated herein by reference. [Background technology]

[0003] In recent years, the world has become increasingly reliant on portable electronic devices that are more powerful, sophisticated, and convenient for a wide range of users. However, while children can readily accept using some aspects of electronic devices designed for more experienced users, young children can benefit from access to interactive electronic devices that are small, lightweight, colorful, playful, informative, ergonomically designed for children (including being child-safe), and easy to use. The systems and methods disclosed herein take advantage of recent advances in the fields of haptic technology, sound generation using small speakers, portable displays, and inertial measurement units (also known as inertial motion units).

[0004] In handheld devices, warnings may be generated by haptic units (also called force sensation transmission) and / or small speakers. Haptic units typically use an eccentric (i.e., unbalanced) rotating mass or a piezoelectric actuator to generate the vibrations that can be felt. Similarly, small speaker vibrations are typically generated using traditional (i.e., associated with larger speakers) electromagnetic moving coil or piezoelectric (so-called buzzer) designs.

[0005] Two-dimensional visual displays consist of any number of monochromatic or multicolored addressable light sources, or pixels. Displays range from a single light source (e.g., an illuminating sphere transmitted through a waveguide) to those capable of displaying a single digit (e.g., a seven-segment display) or alphanumeric characters (e.g., a 5-pixel by 8-pixel array), to high-resolution screens with tens of millions of pixels. Regardless of scale, displays are typically realized as 1) a two-dimensional array of light sources (most often light-emitting diodes (LEDs)), or 2) two sheets of polarized glass sandwiching a liquid crystal material (i.e., forming a liquid crystal display or LCD) that responds to an electric current by allowing various wavelengths of light from one or more illumination sources (i.e., a backlight) to pass through.

[0006] Inertial Measurement Units (IMUs) can incorporate any or all combinations of: 1) linear accelerometers that measure forces generated during motion in up to three axes or dimensions (i.e., forces governed by Newton's second law of motion); 2) gyroscope-based sensing of rotation rate or velocity in up to three axes of rotation; 3) magnetometers that measure magnetic fields (i.e., magnetic dipole moments), including those generated by the Earth; and / or 4) Earth's gravity (including the direction of gravity) by measuring forces acting on an internal mass. IMU accuracy varies significantly depending on size, operating range, compensation hardware used to correct measurements (impacting cost), environmental factors including thermal gradients, the availability of calibration for individual devices, and the (integration) time required to perform measurements.

[0007] Advances in electronics (i.e., hardware), standardized communication protocols, and the allocation of dedicated frequencies within the electromagnetic spectrum have led to the development of a wide variety of portable devices capable of wirelessly communicating with other devices in their vicinity, as well as with larger communication systems, including the World Wide Web. Considerations for which protocol (or combination of available protocols) to utilize for such portable devices include power consumption, communication range (e.g., from a few centimeters to hundreds of meters or more), and available bandwidth.

[0008] Currently, many portable devices use Wi-Fi (e.g., based on the IEEE 802.11 family of standards) and Bluetooth (managed by the Bluetooth Special Interest Group). Less common and / or older communication protocols for portable devices in the home environment include Zigbee, Zwave, IR (infrared), and cellular or mobile phone-based networks. In general (though there are many exceptions, especially considering newer standards), compared to Bluetooth, Wi-Fi offers greater range, higher bandwidth, and a more direct path to the Internet. Bluetooth, on the other hand, has lower power consumption, a shorter operating range (which can be advantageous in some cases), and requires less complex circuitry to support communication.

[0009] Advances in miniaturization, reduced power consumption, and increased sophistication of electronic devices, including those applied to displays, IMUs, and communications, are revolutionizing the portable device industry. Such portable devices are becoming increasingly sophisticated, allowing users to simultaneously communicate, obtain location information, monitor exercise, track health, alert to hazards, capture video, perform financial transactions, and more. Systems and methods that facilitate simple and intuitive interaction with handheld devices, particularly those used by children, would be useful. Summary of the Invention

[0010] Accordingly, provided herein are systems and methods describing lightweight, easy-to-use, and intuitive handheld devices that are particularly well-suited for machine-based interaction by young children. While the devices may be embraced by children, in part, as toys, the computational flexibility built into the devices allows them to be used as a means for machine-based and human interaction (particularly with individuals in remote locations), play, embodied learning, emotional support, communication, creative expression, and imagination enhancement. Furthermore, portable, "fun" handheld devices may stimulate physical movement in children (or adults), including motor movements and kinematic activities.

[0011] Young children may observe older children and adults using mobile devices such as cell phones and tablets for extended periods of time to perform a variety of activities. However, young children typically must first develop the precise motor skills (e.g., touching specific icons on a touch-sensitive display) and the intellectual sophistication to navigate the graphical user interface (GUI) on such devices to achieve a desired goal. Even the concept of pressing one or more simple pushbuttons (e.g., brightly colored buttons) to achieve a desired goal is typically a concept young children must first learn. However, tapping anywhere on the body of a lightweight handheld device or tapping the device against a solid surface, particularly in response to visual, tactile, or auditory cues, can provide children with an intuitive way to use handheld devices in mixed, real, and / or virtual environments without substantial instruction or precise motor skills.

[0012] Determining not only the presence (including timing) of a tap but also its location on the handheld device (including attached components) can be accomplished by classifying electronic signatures from data collected from at least one inertial measurement unit (IMU) embedded within the handheld device. As outlined in the Background section above, IMU data is obtained by sensing a combination of translational acceleration, gyroscopic (i.e., rotational) rate, magnetic forces (including forces exerted by the Earth's magnetic field), and / or gravitational forces (i.e., due to the Earth's large mass). In most multidimensional (up to three-axis) implementations, IMU data can be thought of as a sequence of time-varying vectors containing both magnitude and direction information for the sensed acceleration, orientation, and / or force.

[0013] Fingers (i.e., the digits of either hand) are convenient and intuitive "tools" for tapping on handheld devices. In English, the term "finger" can have an ambiguous meaning. In much of the scientific world, "finger" refers to an appendage of the hand used for manipulation or sensing. However, in some cases (including medicine), the thumb (comprising two bones or phalanges) is considered separate from the fingers (comprising three phalanges) due to differences in size, rigid structure, articulation, and / or function. As used herein, the terms "finger" or (interchangeably) "digit" refer to the protrusion on the hand, including the thumb.

[0014] Similarly, the verb "tap" is used herein to describe an action involving crashing one object into another, accompanied by an instant of contact, a sudden change in acceleration, a change in direction, and / or one or more other discernible cues that indicate contact between the objects has occurred. When contact is made, either object may recoil from the collision process, a sound may be produced, and / or either object may be temporarily deformed in the process. As an English noun, tap has many meanings, including referring to the sound sometimes made when objects are crashed into each other. As used herein, "tap" refers to the overall action or process of crashing two objects into each other. Also, as discussed below, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0015] A tap force is transmitted to one or more embedded IMUs by tapping on a handheld device body that includes fixed components (i.e., forms a solid structure), and such a rigid structure transmits a force in the direction of the tap upon impact (i.e., during the tap). In accordance with Newton's second law, such forces can result in translational motion of the handheld device in any of three dimensions (often designated X, Y, and Z; see FIG. 5 ). Forces can also result in rotational motion of the device (often described as pitch, roll, and yaw; see FIG. 5 ); one way to describe such motion is to use, for example, the geometric center of the device and / or the functional center of the IMU as a reference (i.e., origin). During mathematical analysis, it may also be convenient to consider other locations (e.g., the contact point of the hand holding the device, the contact point of the tap) to locate the origin of coordinate systems and / or vectors representing forces, orientations (e.g., orientation relative to Earth's gravity or gravitational force), accelerations, velocities, and motions.

[0016] A tap may be generated by a user tapping the fingers of the opposite hand (i.e., the hand opposite to the hand holding or grasping the device) on the device, tapping the fingers of the opposite hand (e.g., knuckles, palm) on the handheld device, tapping the fingers of the hand holding the device on the device, tapping another part of the user's body on the handheld device, tapping on the device using a solid object such as a pen or stylus, or tapping the handheld device itself against an object or surface (e.g., a body part such as a knee or wrist, or another object such as an image or text in a page of a book, a desktop, a floor, etc.). A tap may also be generated by tapping a handheld device against another handheld device (e.g., while mimicking combat using swords or sabers). The resulting action for a tap between devices may be specifically limited to, for example, tapping a selected display on one device onto a target display of a second handheld device (e.g., by detecting the tap location) (e.g., during the process of exchanging information between devices and / or device users).

[0017] When impacting a point on a handheld device with a surface, the surface need not be completely rigid. For example, tapping the device on one's knee or the palm of one's opposite hand may cause temporary deformation, at least at skin level, during the tap. As a further example, when tapping a stuffed animal or stretchy toy, the surface being tapped may be flexible. Similarly, tapping on a page in a book or magazine may cause the page being tapped to move slightly.

[0018] As explained in more detail in the detailed description below, the IMU data may undergo one or more classification processes to identify not only the location of the tap on the handheld device, but also other characteristics of the tap (collectively referred to as tap "attributes"), such as measurements of the mechanical properties of the object used to tap, the magnitude and direction of the force applied during the tap, the movement (including magnitude and direction) of the handheld device immediately after the tap, and the orientation of the handheld device during the tap relative to the Earth's magnetic and / or gravitational forces.

[0019] Such differences in sensed acceleration and force under the device user's control can classify the type and / or attributes of the tap, which can be used during control of an activity (e.g., while controlling the sound, display, haptic feedback, and / or other actions of the handheld device itself, or while controlling virtual objects and / or actions displayed on one or more connected devices). Additionally, the timing of the tap, particularly relative to one or more previous taps and / or other events in the device user's environment (including taps on one or more other handheld devices), can factor into additional control features by the handheld device user.

[0020] According to one example, a handheld device for interaction by a device user is provided, the device including: a device body configured to be held in a first hand of the device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and at least one device display attached to the device body operably coupled to the device processor, wherein the device processor is configured to generate a first lighting pattern via the at least one device display, acquire inertial measurement data from the at least one inertial measurement unit, identify an initial tap location by the device user on one of the device body and the at least one device display based at least in part on the inertial measurement data, and generate a second lighting pattern via the at least one device display based at least in part on the initial tap location.

[0021] According to another example, a handheld device for interaction by a device user is provided, the device including: a device body configured to be held in a first hand of the device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and a speaker within the device body operably coupled to the device processor, the device processor configured to generate a first sound via the speaker, obtain inertial measurement data from the at least one inertial measurement unit, identify an initial tap location by the device user on the device body based at least in part on the inertial measurement data, and generate a second sound via the speaker based at least in part on the initial tap location.

[0022] According to yet another example, a handheld device for interaction by a device user is provided, the device including: a device body configured to be held in a first hand of the device user; electronic circuitry within the device body including a device processor; and at least one inertial measurement unit within the device body operably coupled to the device processor, the device processor configured to acquire inertial measurement data from the at least one inertial measurement unit; calculate orientation data and magnitude data from the inertial measurement data; identify an initial tap location by the device user on the device body based at least in part on one or both of the orientation data and the magnitude data; and perform an action at one or both of the device processor and a remotely connected processor based at least in part on the initial tap location.

[0023] Other aspects and features, including the needs and uses of the present invention, will become apparent from consideration of the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0024] A more complete understanding can be obtained by reference to the detailed description when considered in conjunction with the following illustrative drawings, in which like reference numerals refer to like elements or acts throughout the drawings. Examples presented are shown in the accompanying drawings. [Figure 1] Figure 1A illustrates a device user tapping the left-most device display (projecting the letter "C") using the index finger of the hand opposite the hand holding the handheld device as a means of interaction. Figure 1B continues the scenario illustrated in Figure 1A and illustrates a machine-based interaction where the index finger of the hand opposite the hand used to grasp the handheld device taps the center device display (projecting the letter "A"). [Figure 2]FIG. 2 illustrates tapping the upper body of a handheld device, including the area containing the device speaker, as a means of interaction (e.g., in response to interactive sounds produced by the speaker) using the thumb of the same hand used to hold the handheld device. [Figure 3] Figure 3A illustrates a device user interacting by tapping the center display (projecting an image of an aquatic animal) of a handheld device against a solid object (e.g., a table or desk surface), while Figure 3B illustrates a human-machine interaction scenario similar to that shown in Figure 3A, where the right-most display (projecting phonetic symbols) is tapped against a solid object (e.g., a table or desk surface). [Figure 4] Figure 4A shows exemplary sampled data and calculated acceleration magnitude traces from three IMU accelerometer channels while a finger is tapping on the left display of a handheld device (see Figure 1A), illustrating the method used to detect the occurrence and location of the tap. Figure 4B shows exemplary acceleration data and calculated acceleration magnitude traces while a finger is tapping on the right display of a handheld device, illustrating acceleration differences identified to help identify the tap location (compared to Figure 4A, which shows the trace when the left display is tapped). Figure 4C shows exemplary acceleration data and calculated acceleration magnitude traces while a handheld device's right display is tapped against a desktop, illustrating the differences in the traces (used to identify tap mode or attribute) between tapping the display against a solid surface and tapping the right display with a finger (e.g., compared to the trace shown in Figure 4B). [Figure 5] Figure 5 shows example coordinate systems and rotation axes that may be used to describe various dimensions of motion, velocity, and acceleration as elements of the process of calculating tap position on a handheld device from acquired IMU data. [Figure 6]FIG. 6 is a flowchart outlining exemplary steps for processing and locating tap-based interactions after projecting a lighting pattern onto the display of one or more handheld devices. [Figure 7] FIG. 7 is a flowchart outlining exemplary steps for processing and locating tap-based interactions following a sound generated by a speaker or buzzer on a handheld device. DETAILED DESCRIPTION OF THE INVENTION

[0025] Before describing the examples, it is to be understood that this invention is not limited to particular examples described herein, as such may, of course, vary. Also, it is to be understood that the terminology used herein is for the purpose of describing particular examples only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. It should be noted that in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, a reference to a "compound" includes a plurality of such compounds, and a reference to a "polymer" includes a reference to one or more polymers and their equivalents known to those skilled in the art.

[0027] Where a range of values ​​is given, each intervening value between the upper and lower limits of that range, to the tenth of the unit of the lower limit, is understood to be specifically disclosed unless the context clearly dictates otherwise. Each smaller range between any stated value or intervening value in a stated range and any other stated value or intervening value in that stated range is encompassed within the invention. The upper and lower limits of such smaller ranges may each independently be included or excluded, and whether both or neither are included in the smaller range, each range is encompassed within the invention, subject to any specifically excluded limit in the stated range. When a stated range includes one or both of the limits, ranges excluding either or both of those included limits are also intended to be included within the invention.

[0028] In this specification, certain ranges are presented with the term "about" before the numerical values. The term "about" is used herein to provide literal support for the exact numerical value that follows the term, as well as a numerical value that is close to or approximate to the numerical value that follows the term. In determining whether a numerical value is close to or approximate to a specifically stated numerical value, the unstated near or approximate numerical value may be a numerical value that, in the context in which it is presented, provides a substantial equivalent to the specifically stated numerical value.

[0029] As discussed in the Summary of the Invention, a tap can be generated by intentionally moving an object (i.e., a "impact object") and then striking it against a location on the surface of the handheld device (i.e., a "tap location") selected and / or targeted by the user. The impact object can be, for example, the fingers of the opposite hand, another body part, the fingers of the hand holding the device, a stylus, a stick, a pencil, etc.

[0030] A tapping force can also be generated by impacting (i.e., intentionally moving) a target location or area on the surface of a handheld device against a solid surface (or an at least partially rigid surface, such as human skin, a stuffed animal, or the page of a book). Combining such movements can simultaneously move both the handheld device and another object (e.g., a child's toy, a hand) toward one another. Generally, the relative movement of the handheld device with respect to the surface being tapped determines the tap characteristics (e.g., peak force, acceleration, calculated tap location) versus which object was moved (e.g., relative to the ground or other objects in the user's environment). Pre-tap and post-tap IMU data streams may be useful in determining whether a striking object was used to tap a stationary device, whether the device was forcibly moved toward another object (e.g., see Figure 4C), or whether both processes occurred simultaneously.

[0031] As mentioned in the Background section, the IMU data stream is 1. Accelerometer data with up to three channels (representing three spatial dimensions denoted X, Y, and Z; see Figure 5 ); 2. Gyroscope rotation rate and rotation rate for up to three channels (i.e., representing rotation around three axes; see Figure 5). It may consist of one or more of:

[0032] Both the acceleration data and the gyro data can be expressed as one or more time-varying vectors using, for example, Cartesian, polar and / or spherical coordinate systems (see FIG. 5).

[0033] Optionally, a data stream indicating the orientation of the handheld device relative to Earth's gravity and / or magnetic or gravitational forces can be considered. Such orientation data can be a factor in the process of identifying the tap location. For example, if the device is held in the hand by the device user "upside down" (in any of up to three spatial dimensions) (or at least not held as intended), a transformed coordinate system reflecting the measured or anomalous device orientation can be used to consider the expected acceleration and / or gyroscope data stream. Furthermore, the enumerated potential tap locations can be redefined based on the likelihood that one or more surfaces of the handheld device become more accessible while other surfaces become obscured as a result of such repositioning by the hand holding the device.

[0034] Device orientation data can also be used to represent the movement of the handheld device relative to previous movements of the device (e.g., when detecting continuous rotations of the device similar to turning a conventional knob) and / or relative to other objects in the device user's environment, such as a viewable display screen. In additional examples, if the orientation (e.g., orientation relative to gravity) of a vertically oriented display screen is known and the IMU data stream includes the direction of gravity relative to the body of the handheld device, a displayed image of the movement of the handheld device (or any virtual object controlled by the device, such as a doll or virtual toy) can be displayed in the same orientation as the physical device or from another selected perspective. In a further example, the orientation of the device during a tap (or orientation relative to the Earth's magnetic field or gravity) can be used to define one or more tap attributes (described below) that can be used as factors to control functionality of the handheld device by controlling or adjusting actions on the device itself and / or a connected device (e.g., a connected tablet).

[0035] Identifying tap locations on a handheld device can be viewed as a classification process that uses patterns (e.g., comparison with templates) or “signatures” in the IMU data stream to calculate the most likely tap locations. The first step in such a classification process is to identify one or more ways to designate potential tap locations on the surface of the handheld device. One way to indicate potential tap locations is to cover the device surface with a virtual mesh (e.g., composed of triangles, finite elements) or rectangular grid pattern (e.g., in a curved two- or three-dimensional space), where the intersections of the mesh or grid may indicate potential tap locations. This process effectively assigns a uniform spatial distribution of potential tap locations to the surface of the handheld device. One drawback of this approach is that taps at two or more closely spaced grid locations may not produce tap signatures that are distinguishable from one another (i.e., uniquely classifiable) (at least within the variability of, e.g., different users, different tap intensities, different hand grips, etc.).

[0036] Another exemplary method for identifying tap locations includes enumerating a desired number of different locations on the device surface. In effect, the enumeration aggregates multidimensional locations on the device into a single target classification dataset, with the number of potential tap locations being related to (i.e., effectively defined by) the spatial resolution of the tap locations within the region. Such an enumeration scheme allows for varying the density of potential tap locations across the device's surface, thereby allowing for lower spatial resolution, for example, when identifying tap locations in some regions of the device (e.g., areas typically covered by hands) and / or when using applications that may benefit from more closely spaced tap locations (e.g., regions surrounding one or more displays; see FIG. 1A ).

[0037] As an example among various applications, when responding to a simple query, only three tap locations (e.g., on each of the three displays, see FIGS. 1A and 1B) may be required, while other applications may utilize or attempt to identify dozens of different tap locations across various surfaces of the device. In addition to tap locations on one or more fixed displays (e.g., see FIGS. 1A and 1B), tap locations may include the area of ​​a speaker or buzzer (see FIG. 2), any location on the top surface of the device body, an edge or corner of the device, or surfaces on the sides or edges of the device body or any of the external elements mentioned above.

[0038] Analog-to-digital (A / D) conversion techniques known in the art can be used to convert the analog IMU data into a digital format suitable for numerical processing. IMU sample rates typically range from about 100 samples / second to about 10,000 samples / second, with higher IMU sample rates resulting in tradeoffs in signal noise, cost, power consumption, and / or circuit complexity (as further described in the Background section above). Because taps are relatively fast events (e.g., typically lasting in the range of about 10-100 milliseconds), higher IMU A / D sample rates enable the identification of more distinct tap locations on a handheld device.

[0039] In embodiments, the IMU sampling rate can be dynamically changed. For example, a slower sample rate can be used in most cases to conserve power and computational resources (e.g., when identifying gestures based on hand movements other than taps). As soon as an increase in signal (i.e., a signal indicative of a tap) is detected in the accelerometer and / or gyroscope data stream, the A / D sample rate can be increased to the maximum available in the IMU device. Alternatively or additionally, the high rate sampling can be buffered (e.g., held in a circular buffer) by hardware or firmware. Upon detection of a potential tap, one or more processors can acquire samples in the buffer prior to the detection of the tap, thereby, for example, more accurately assessing the earliest time of the tap (e.g., extrapolating one or more signals exceeding a threshold to a baseline) and including additional samples to classify the acquired data as a tap location. In a further alternative or additional example, the hardware can perform threshold detection (see FIGS. 4A, 4B, and 4C) and suspend execution in one or more processing elements (i.e., running firmware or software) each time a tap is detected by a hardware element.

[0040] Broadly speaking, the classification process for converting IMU data into tap locations involves two general approaches: 1) numerical approaches involving multi-channel template matching and / or frequency-domain (e.g., Fourier transform) analysis; and 2) neural network approaches that train a network using datasets of taps of known locations collected under a variety of conditions (e.g., different users, different fingers or other objects used to tap, different hand sizes and / or positions holding the device, different tap forces, and different device orientations). Such training datasets can be obtained from individual users (or a small number of users) with a limited range of forces and / or tap locations (e.g., due to age or pronunciation ability). Alternatively or additionally, a wide range of users can be used to generate a neural network training dataset (e.g., specifically including taps using fingers) that can then be applied globally (i.e., to any device user).

[0041] Numerical approaches can utilize a computational model of the handheld device, combined with the location of contact by the hand holding the device. From such considerations, the expected magnitude, timing, and direction of force applied during taps at various locations on the device can be estimated. Template matching techniques can then be applied to all or a portion of the IMU data stream. For example, a template waveform (e.g., modeled and / or empirically obtained) from each potential tap location can be compared to the acquired data to calculate a correlation coefficient (or similar measure). The template waveform associated with the location with the highest correlation corresponds to the most likely tap location. Alternatively or additionally, simply comparing the sequence of positive and negative peaks in the acquired data (e.g., across all available IMU channels) with those in the template waveform may be sufficient to uniquely identify the tap location, especially when the number of target tap locations is small (e.g., based on the interactive activity being performed).

[0042] During such calculations, it may be useful to use one or more polar and / or spherical coordinate systems to more easily consider three-dimensional (e.g., rotational) velocity and acceleration vectors, for example, centered around the functional center of the IMU and / or a contact point including the hand holding the device. It may also be useful to transform the time-based IMU data stream into the frequency domain (e.g., processed using a multidimensional Fourier transform), thereby determining the presence of characteristic frequency components and phase differences (i.e., related to time and / or sequence) between frequency components in the data stream, which may help distinguish between different tap positions.

[0043] A neural network approach may involve, for example, using the IMU data stream as input to one or more neural networks that include either a single output identifying the most likely tap location (e.g., indicating an enumerated tap location) or a binary (e.g., yes / no) output associated with each location. Either neural network configuration may be further configured to provide a confidence level (or equivalent measure) associated with the degree of match of each potential tap location. The neural network may also be trained to output one or more tap attributes, described in more detail below. Tap attributes may include, for example, the strength of the tap, the orientation of the handheld device, the movement of the device at the time of the tap, whether the tap was made with a soft or hard object, whether the tap was made with the fingers of the opposite hand (relative to the hand holding the device), etc.

[0044] Techniques known in the art can be used for digital signal processing (DSP) of such time-varying data streams, where 1) multilayer perceptrons (MLPs) and 2) recurrent neural networks (RNNs) are example network topologies (i.e., suitable for processing time-based data streams). Typically, backpropagation can be used to train such networks. Optionally, the neural network can be made adaptive by noting when modifications are made in the virtual activity controlled by the handheld device. For such adaptive modifications, the neural network can be further trained using a supervised learning approach.

[0045] Depending on factors such as the IMU sample rate, noise in the IMU data, the handheld device's mechanical design, the computational approach, and the temporal / mechanical consistency of taps generated by the user (or other users of the device), the algorithm that converts IMU data to tap position can target any number of distinct tap locations. Tap locations on the handheld device can be used as input to control or coordinate resulting "actions" in the real and / or virtual worlds. Such actions can be 1) strictly confined to the handheld device, such as a haptic vibration, sound, or symbol displayed on the handheld device; 2) communicated to another device in the device user's environment (e.g., a sound generated by a nearby speaker or an image displayed on a nearby screen); and / or 3) communicated to a remote device (i.e., handheld or other device) where the tap and / or its location can result in a visual, audio, and / or other indicator for a remotely connected individual.

[0046] Additionally, as will be described below, the time of occurrence of the taps (including the interval between taps) can further be used as a factor in controlling or adjusting the action resulting from the taps. Numerous methods can be used to determine the time of occurrence of the taps, including: 1. The time at which an acceleration or velocity component exceeding a threshold was first detected, optionally including a technique for extrapolating back to a baseline from multiple samples collected before the threshold was exceeded as a means of improving the accuracy of the initial time estimate; 2. The time of maximum (i.e., peak) acceleration or velocity, where multiple sample points (e.g., fitting to a parabola) can be considered to increase the accuracy of determining such peaks; and / or 3. A midpoint (or other selected reference point) between first exceeding the threshold level and returning below the threshold, which may include extrapolation techniques (e.g., linear fitting) that consider multiple samples within the threshold region for increased accuracy.

[0047] In addition to using the time between taps as one input for controlling or adjusting the resulting action, the time between a "stimulus" such as a haptic vibration, sound, or symbol displayed on the handheld device, or other stimulus sensed in the device user's environment (e.g., a played sound, an image displayed on a distant screen), and the response tap time can optionally be used as an additional modulator of the action. Such tap timing commands can be intentional (i.e., the device user intentionally sets the timing at which one or more tap responses occur) or unintentional (i.e., the stimulus-response time is measured by the device in a manner transparent to the user). The resulting action controlled or adjusted at least in part by the stimulus-response tap timing can be within the handheld device (e.g., display of one or more symbols, generation of one or more sounds, haptic vibration), and / or the tap time can be transmitted to another device (e.g., a tablet, laptop, e-book) where the resulting action is performed or adjusted at least in part by the tap-timing measurement generated on the handheld device.

[0048] In additional exemplary configurations, two or more handheld devices can be used (e.g., by two or more co-located individuals or two or more remote individuals connected via telecommunications) to measure the time between taps. For example, one device user can initiate an action in a virtual environment by tapping on a first handheld device. Within a preset time, a second user can complete the virtual action by tapping on a second device. Measuring the time between taps and / or other control features using two or more handheld devices can be performed by any number of device users. Such multi-user shared control using taps is particularly useful during gameplay activities or other shared activities within a virtual environment. Such shared control of activities (including the use of handheld devices) is described in U.S. Patent No. 11,334,178, filed August 6, 2021, and application Ser. No. 17 / 531,571, filed November 19, 2021, the entire disclosures of which are expressly incorporated herein by reference.

[0049] Optionally, in additional examples, the orientation of the device can be used as an input to control or adjust the resulting action. The orientation of the device can be determined from an IMU component that senses the direction of the Earth's gravity and / or magnetic field, as described above. As an example, a tap performed when the handheld device is held approximately horizontal can be used to indicate a "no" response by the device user during an interaction, while a tap performed when the handheld device is held vertically can indicate a "yes" response. Furthermore, the orientation of the handheld device can be used over a continuous range of orientations (e.g., partially up or down, similar to the orientation of a clock hand) during one or more taps. For example, the orientation of the handheld device can be used (over a continuous range) to select a particular hue or color while drawing.

[0050] Similarly, in a further example, tap force (e.g., as determined from the amplitude of magnitude peaks, particularly within the IMU data stream (see, e.g., FIGS. 4A, 4B, and 4C)) can be used to convey a continuous range of control or adjustment of the resulting action. For example, as described above, if device orientation is used to indicate a "yes" or "no" answer, a strong tap can be used to indicate a strong degree of confidence by the device user that the answer is correct. Conversely, a weak tap can be used to indicate uncertainty about the answer.

[0051] In addition to peak force, other waveform characteristics or “signatures” in the IMU data stream can be used to identify the source of a tap and to control or adjust the resulting action. Such signatures from IMU data include the direction, duration, frequency content, and phase difference of the tap and may help identify, for example, the hand or specific finger used to generate the tap. For at least a small group of device users, tap characteristics can distinguish which user is holding the device based on factors such as hand size and how firmly the device is held (i.e., identifying the device movement resulting from the force applied during the tap) and / or how the device is grasped in large versus small hands (i.e., resulting in different contact points that constrain the device movement during the tap). In a further example, tap signatures can distinguish that a stylus or pencil (as opposed to a finger) is being used to tap the device.

[0052] As yet another example, tapping a device on a hard, immovable surface, such as a desk or floor, can be distinguished from more amenable surfaces, such as a book page or stuffed animal. Tapping on a rigid structure generally generates a tap force (sensed by one or more IMUs) that is shorter in duration, has a higher peak amplitude, and contains a higher compression wave frequency component throughout the tap compared to tapping on soft tissues on most body surfaces (see Figure 4C). Conversely, tapping on an object that may be somewhat "flexible" during the tap generally generates a force and resulting acceleration (measured) that is longer in duration, has a lower peak amplitude, and contains a lower compression wave frequency.

[0053] The physical location of the handheld device and its orientation in (three-dimensional) space can be further determined by camera-based tracking of the handheld controller. The camera-based measurements of the handheld controller (e.g., orientation, position, velocity, acceleration within the camera's field of view, including orientation relative to other objects in the field of view) can be combined with the IMU-based data stream to provide further control or adjustment of the resulting action. Systems and methods for determining such camera-based measurements are described in U.S. Patent No. 11,334,178, filed August 6, 2021, and Application No. 17 / 531,571, filed November 19, 2021, the entire disclosures of which are expressly incorporated herein by reference.

[0054] In summary, the identification of a tap and its location on a handheld device can be considered just one of many control characteristics (e.g., "tap attributes") that can be used to direct or coordinate (i.e., affect) an action. The following list summarizes measured attributes of a handheld device's tap that can facilitate interaction with actions and / or activities embedded within the handheld device and / or can be transmitted to other electronic devices (e.g., tablets, laptops, e-books, or magazines). Tap attributes can be used to interact with the real world (e.g., to control IoT components such as light switches or thermostats) and / or within virtual environments (e.g., to control a cartoon-like avatar or the turning of pages in a virtual book).

[0055] 1. The presence and time of a tap, especially when associated with the occurrence of other events in the device user's environment, can be used as a primary control feature. As a visual example, a tap (either on a handheld device) can be used to control the time to turn a page in a virtual book displayed on a connected tablet device. As an auditory example, the presence of a tap can be used to indicate a selected answer immediately after articulating the answer in a series of possible answers played using a speaker (e.g., on the handheld device or a nearby electronic device).

[0056] 2. The identified location of a tap on a handheld device can be used to control or coordinate actions, particularly when making so-called "one-of-N" selections. As an example, the surface of the handheld device can be "mapped" to the appearance of a human figure, avatar, animal, or other object displayed on a connected screen. In this example, the top, bottom, and sides of the handheld device can correspond to the top, bottom, and sides (i.e., a similar spatial arrangement) of the object displayed on the screen. Tapping a specific location on the handheld device can indicate that subsequent movement of the handheld device will be reflected in the movement of the component selected by the tap (e.g., a doll's head or arm).

[0057] 3. The force or intensity applied to a tap can be measured (e.g., within the peak amplitude of acceleration). For example, under the control of a handheld device, one can distinguish between a light tap and a stronger tap to indicate confidence in the answer or urgency to take action.

[0058] 4. In addition to the overall force applied during a tap, the (more subtle) timing of various force characteristics can also be used to identify or distinguish the characteristics of the object striking the handheld device. For example, a pencil or stylus strike can be distinguished from the use of (e.g., relatively soft) fingers of the opposite hand. Subtle movements before the strike can distinguish between the use of fingers of the opposite hand and the use of fingers of the hand holding the device. Furthermore, considering the movement of the device over time can distinguish between an object being moved to strike the device and the handheld device itself moving to strike another surface.

[0059] 5. The interval between two taps is typically precisely controlled and then measured by the device user. As a simple example, distinguishing between long and short taps provides a simple method for binary (e.g., "yes" and "no") control. The continuous nature of measuring such intervals (at least up to the ability of a typical user to precisely dictate time intervals) is considered particularly useful. As an example, the interval between taps can be used to control screen brightness (e.g., over a continuous range of screen illumination).

[0060] 6. Identifying the temporal placement of multiple taps and / or patterns of taps provides even broader temporal control features. Meaning can be imparted to tap sequences by comparing the intervals between taps to a temporal reference (e.g., 1 second, 2 seconds, etc.) or to each other (e.g., a short tap followed by two long taps). Extending the concept of conveying control through tap patterns significantly, taps can also be used to represent musical melodies, preassigned interpretations (e.g., when setting a clock, long intervals increase units of time and short taps increase minutes), or even Morse code.

[0061] 7. Handheld device movement between (and during) taps can provide additional attributes during device-enabled control functions. Both the magnitude and / or direction (including velocity and / or acceleration) of movement during and between taps can be considered. As a simple example, the direction of movement preceding or following a tap can be used to indicate whether a page in a virtual book should turn forward or backward. In a further example that considers both magnitude and direction of inter-tap measurements, if an object on the screen is the focus of attention (e.g., a person speaking), a nearby displayed object can be designated as the new focus by the user of the handheld device by indicating the distance and direction (relative to the previous focus position) between two taps.

[0062] 8. The orientation relative to the Earth's magnetic field and / or gravity during tapping can provide additional tap attributes. As an alternative example, tapping while holding the handheld controller vertically can indicate approval of a selection (e.g., a selection shown to another person, visualized on a screen, or spoken through a speaker), while tapping while holding the device approximately horizontally can indicate disapproval. As a further example involving a continuous range of control utilizing device orientation, holding the handheld device vertically and then rotating and tapping (e.g., while visualizing a traditional clock face) can control the intensity of a connected light (e.g., via IoT). If the orientation of one or more other objects (e.g., relative to Earth's gravity) is known, the orientation of the handheld device relative to those objects can be calculated. In other words, Earth's gravity can provide a common reference direction for both the handheld device and the other object. As an example, if the display screen is oriented vertically, the three-dimensional orientation of the handheld device at the time of the tap can be used to calculate the viewpoint of a three-dimensional representation of one or more items depicted on the screen (e.g., including an image of a doll or the handheld device itself).

[0063] 9. Tap attributes generated on one handheld device can be combined (e.g., in real time) with tap attributes generated on one or more additional handheld devices. The additional devices may be co-located with the device user and / or used by interconnected users located remotely. The second handheld device may also be controlled by a second hand of the device user.

[0064] When considering the manipulation of these handheld devices as an ensemble, multiple tap attributes can be used in various combinations to intuitively and effectively generate multiple real-time controls within a virtual environment, thereby also controlling aspects of the real-world environment. As an example of the latter, a controller can be used to adjust a multi-speaker sound system, and when a speaker volume control function is selected, the tap location on the handheld device can be used to specify which speaker or combination of speakers to control. Multiple speakers can be specified by tapping two or more times quickly at different locations on the handheld device. The device can then be rotated in a forward plane to control the volume of one or more speakers (e.g., similar to a volume control knob), and this in-plane and out-of-plane movement can control the balance of bass and treble.

[0065] In a further example of using simple, intuitive controls suitable for young children within a virtual environment, a handheld device can be used to control various aspects of a doll (or other character, avatar, or object) displayed on one or more connected display devices (which may also include remotely connected display devices viewable by others). A single tap on the body of the handheld device can specify that the device's movement (rotational and / or translational) is reflected in the doll's entire body movement. A single tap on the side of the handheld device can indicate a desire to control the movement of a particular limb. Multiple closely timed taps can be used to indicate a desire to control multiple limbs simultaneously through movement of the handheld device.

[0066] As noted above, spatial resolution or "positional resolution" (i.e., the number of distinct locations identified by the classification process) can vary within different applications, from, for example, just a few tap locations (e.g., when responding to yes / no questions) to dozens or more distinct tap locations (e.g., when identifying various anatomical locations and outfits or actions controlled by a doll). The number of potential locations can also vary from tap to tap. For example, an answer to a series of questions in a quiz might be selected from one of three potential answers (e.g., each indicated by a different tap location) to address the first question, while the next question in the quiz might be selected from six potential answers (each potential answer, again, associated with a different tap location).

[0067] In additional examples, classifying a dynamically changing number of locations can be addressed by using separate classification processes (e.g., separate neural networks trained on taps at different numbers of potential tap locations) or by combining the outputs of the classification processes to "lump" two or more classified locations into a single result. The latter approach can further lump or combine confidence levels (or other statistical measures) to identify a single, most likely cumulative tap location or region. Such confidence levels can, for example, be summed or accumulated to form a lumped or local confidence level. In some outlier cases using this approach, the cumulative probability of a collection of two or more classified locations with moderate certainty may exceed a single location with a high level of certainty, but it is surrounded by classified locations with low levels of certainty.

[0068] In a further example, the use of tap locations and / or signaling actions (e.g., following a tap) by a device used to control a virtual action or object can take interactive context into account, potentially leading to the concept of “interpretive control.” “Interpretive control” can relax constraints on one or more tap locations, one or more orientations, and / or one or more movements specified by one or more controllers based on the context. For example, if a response to a visual or auditory query is intended to be based on tapping one of three displays, and the correct response is indicated by tapping the display on the left side of a handheld device (see, e.g., FIG. 1A ), interpretive control would allow a tap (e.g., by a young child, particularly) at any location on the left side of the handheld device to be interpreted as the correct answer.

[0069] Interpretive control can reduce tap location precision, timing constraints, repetition rate, and / or the number of degrees of freedom required of the control device by assuming user intent based on the interactive context. Examples of context interpretations that may lead to relaxed controller precision include specifying a choice from a limited number of possible choices, using one or more previous and / or frequent choices to place a choice in a more easily accessible tap location (e.g., tapping on a display as shown in Figures 1A and 1B), and utilizing tap patterns that tend to indicate similar choices (e.g., a double tap at a location indicates that a previous tap pattern at a selected location should be repeated).

[0070] Interpretation control not only executes or adjusts virtual activities based on tap location, but can also be applied based on tap attributes, such as device orientation or movement during or immediately after the tap. For example, device movement during and / or after a tap can be used to hammer a virtual nail (e.g., displayed on a connected display device). Interpretation control allows any device movement (e.g., in any direction) following a tap to result in a video sequence of the nail being hammered (e.g., there is no requirement regarding the direction of the nail head and no minimum movement speed for the nail to move through the virtual board). Furthermore, interpretation control can be used to fully drive a nail into a virtual board with only a limited number of hammer strokes (e.g., three). Similarly, rotation of a handheld device following a tap can be used to control a virtual screwdriver, where any rotation angle can result in a full rotation of the screw, or a limited number of rotations (similar to driving the nail described above) can fully insert the screw.

[0071] Interpretation control may be particularly useful in interactions involving young children, the elderly, or individuals with reduced motor and / or cognitive function. Further aspects of "interpretation control" are described in more detail in U.S. Patent No. 11,334,178, filed August 6, 2021, and co-pending application Ser. No. 17 / 531,571, filed November 19, 2021, the entire disclosures of which are expressly incorporated herein by reference. Determining context from audiovisual content and then generating interpretation control based on such context is described in more detail in U.S. Patent No. 11,366,997, filed April 17, 2021, the entire disclosures of which are expressly incorporated herein by reference.

[0072] In further examples, “bimanual control” of one or more virtual objects can be performed by combining the ability to specify a location on a touch-sensitive display using one or more fingers of one hand (or using one or more pointing instruments, such as a stylus) with the near-simultaneous generation of additional activity control and / or coordination capabilities via a handheld device using a second hand. Bimanual control, combining the location identified on the touch-sensitive display with measured tap characteristics (e.g., tap location, timing, and / or attributes) on the handheld device, can create multiple degrees of freedom while controlling a virtual object or activity. For a single user, touching the touch-sensitive display with the fingers of one hand may limit the ability to tap the handheld device to using the fingers of the hand holding the device and / or tapping the device against a solid surface. Aspects of bimanual control by a single device user are described in more detail in U.S. Pat. No. 11,334,178, filed August 6, 2021, the entire disclosure of which is expressly incorporated herein by reference.

[0073] Bimanual control can also be performed by one user tapping with their fingers (or other tapping mechanism) on the handheld device while a second user simultaneously designates a location on the touch-sensitive screen. Such control functions by two separate users (who may be communicating remotely via telecommunications) can collaboratively perform and / or coordinate virtual actions and activities. Bimanual collaborative control performed by two device users is further described in co-pending application Ser. No. 17 / 531,571, filed Nov. 19, 2021, the entire disclosure of which is expressly incorporated herein by reference.

[0074] In additional examples, portable electronic devices, while not strictly "handheld," are attached to or operated by other parts of the human body. A device whose tap location is determined based on an IMU data stream can be attached to an arm, leg, foot, or head, for example. Such placement can be used to address accessibility issues for individuals with limited upper limb and / or hand movement and / or no hands, and / or in situations where hands are required for other activities. The timing and location of taps can be tracked based on tapping motions while held or generated by other body parts.

[0075] The handheld device may further include one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, one or more cameras, an optical cardiac sensor, and an electrical cardiac sensor, each operatively coupled to the device processor. In combination with the location, timing, and attributes of taps, these components may provide additional means for a user to control actions using the handheld device. Additionally, a battery powering the electronic components may allow the handheld device to operate independently of other power sources.

[0076] In yet another example, detecting tap locations based on IMU data streams may largely (but not necessarily) eliminate the need for push buttons and / or other forms of (e.g., finger-based) touch control, although the absence of physical switches does not necessarily mean the appearance of touch structures will also be eliminated. Indeed, images of push buttons or other symbols can be added (e.g., printed) to the surface of the handheld device at various potential tap locations. For example, particularly for young children, various tap locations can be indicated by brightly colored circles (or other shapes) on the surface of the handheld device. Optionally, such symbols can be added using temporary adhesives (including as so-called “stickers”) and can be swapped out for different user applications or simply for fun as the child grows older and / or develops different personal preferences. Such adhesives and / or fastening components can be elements of various skins and / or other accessories that can be used to “dress up” the handheld device (e.g., as a doll, puppet, or toy car).

[0077] 1A and 1B show an exemplary tap sequence that allows a device user to indicate a selection from available responses displayed on displays 12a, 12b, and 12c of handheld device 10. Specifying such a response may be included, for example, in a machine-based interaction during an educational session designed to help young children learn to spell and / or pronounce words. In this example, the algorithm used to classify the tap location is informed that possible outcomes are limited to three locations on the device display (a small number compared to typical applications, potentially simplifying the classification process).

[0078] FIG. 1A shows a handheld device 10 being held in a device user's right hand 14a (the remainder of the user's body is not shown). The handheld device includes three spherical components 11a, 11b, and 11c that may be used as elements of camera-based tracking (not shown) of the device. Each of the three spherical components 11a, 11b, and 11c includes a display 12a, 12b, and 12c, which separately project the letters "C" on 12a, "A" on 12b, and "T" on 12c, which together form the word "CAT." A grille covering a speaker 13 within the handheld device is also visible from the vantage point of FIGS. 1A and 1B.

[0079] During a session using the electronic device, the infant is asked to find the first letter C in the word "CAT." Alternatively or additionally, the infant is asked (e.g., via speaker 13) to indicate the letter that produces the phonetic "k" sound. As shown in FIG. 1A, the child uses the index finger 15 of their left hand 14b to strike or tap (via an up-and-down motion 16a) the top of a sphere 11a containing a display 12a projecting the letter "C." Which sphere was tapped is then classified based on the acquired IMU data to determine whether the answer the child selected is correct.

[0080] As shown in FIG. 1B, the child is then asked to find the sound associated with the next letter "A" and / or phonetic symbol "ae." The correct answer can be indicated by tapping anywhere on the central sphere 11b of the handheld device 10. FIG. 1B shows the child tapping the central sphere 11b using the same up and down motion 16b of the index finger 15 of their left hand 14b. Once tapped, a classification process is performed based on the acquired IMU data to determine which sphere was tapped, thereby determining whether the answer indicated by the tap is correct.

[0081] Figure 2 illustrates the ability to direct tap locations on a handheld device 20 using one hand 24. In this case, handheld device 20 is shown held in the device user's right hand. The palm of the right hand (obscured by the device in Figure 2) and fingers 25b, 25c, 25d, and 25e, excluding thumb 25a, can mechanically stabilize the entire device 20, thereby enabling movement of right thumb 25a (relative to device 20) to, for example, reach the device speaker 23 and three spherical, camera-based tracking components 21a, 21b, and 21c. Each of the three spheres includes a display 22a, 22b, and 22c.

[0082] The speaker 23 and the area of ​​the three spheres 21a, 21b, and 21c are easily accessible to the thumb 25a. For most people (e.g., depending on manual dexterity), the area along the left side 27a of the device 20 and most of the top surface 27b can also be tapped by flexing the thumb 25a. Individual fingers 25b, 25c, 25d, and 25e can also be used to tap the sides of the device 20. It is also possible to tap the backs of the displays 21a, 21b, and 21c, the right side of the handset (not shown in the field of view of FIG. 2), and / or the bottom of the handheld device 20 (not shown) by rotating the device in one's hand (i.e., still using one hand) before tapping.

[0083] A user can indicate a selected response by tapping with thumb 25a in response to an auditory prompt using device speaker 23 and / or a visual prompt projected by any of displays 22a, 22b, 22c. As shown in FIG. 2, a simple up-and-down movement of the thumb results in a tap in the area of ​​speaker 23. Alternatively or additionally, a device user can indicate a response and / or selection by tapping multiple times 26 in approximately the same area of ​​the device. Once tapped, a classification process can be performed based on acquired IMU data to determine which area of ​​the device was tapped and / or the timing of multiple taps.

[0084] 3A and 3B illustrate the movement of handheld device 30 itself to slap or tap selected areas of the device against an object or surface 37. Handheld device 30 is shown held by the palm (obscured from view by the device) and fingers 35b, 35c, 35d, and 35e, including thumb 35a, of a user's right hand 34. Three spherical elements 31a, 31b, and 31c attached to the device each contain a display 32a, 32b, and 32c. The leftmost display displays the word "dog" 32a. The center display 32b displays an image of an animal, and the leftmost display 32c displays symbols representing various sounds. A perforated cover covering an embedded speaker within device 30 can also be seen within the field of view of FIGS. 3A and 3B.

[0085] In FIG. 3A , a device user may wish to indicate that central display 32b showing an animal is a selected response in response to a visual prompt (e.g., on display 32b) or an auditory cue (e.g., via speaker 33). The user may indicate this response by tapping 36a on the bottom of central sphere 31b against a solid surface, such as the edge of a table or desk 37. A classification process to determine which sphere was tapped by the tap (e.g., including identifying that the bottom of the display was tapped) may be performed based at least in part on the acquired IMU data. The classification process may also distinguish taps of device 30 against a hard object (e.g., based on a large peak amplitude and short duration of acceleration during the tap) from taps generated by a finger (i.e., smaller movement and / or acceleration forces generated by a softer contact surface).

[0086] 3B, the desired response to the subsequent prompt may include tapping the right-most display 31c (e.g., displaying symbols representing a phonetic sound) against a solid surface 37. In this case, the handheld device 30, including the three displays 31a, 31b, 31c and the speaker 33, is rotated slightly by the device user's right hand 34 before performing an up-and-down tapping motion 36b against the corner of a table or desk 37. Upon being tapped, the classification process can determine, based on the acquired IMU data, that the outer edge of the right-most display sphere 31c was tapped against the solid surface 37, and that the device was moved (sideways) to effect the tap.

[0087] Figures 4A, 4B, and 4C show examples of IMU accelerometer data acquired while tapping on a handheld device, illustrating some of the distinctive features in the data stream generated by various tapping styles and tapping locations. Each plot shows data from a 3-axis accelerometer, with three orthogonal axes (labeled X, Y, and Z) oriented approximately along the axes shown in Figure 5. Accelerometer measurements for each axis were sampled at approximately 6,667 samples per second. Also shown is the acceleration magnitude |A|, which is calculated by treating the accelerometer data as a 3-dimensional vector using the following formula: TIFF2026500003000002.tif9170where, X i , Y i , Z i denotes the accelerometer sample in each of the three dimensions ("i" denotes the sample index), and X b , Y b , Z b Figures 4A, 4B, and 4C show so-called "baseline" values ​​in each of the same three dimensions. Baseline values ​​can account for factors such as electronic offset of the samples and can be determined during periods of no accelerometer motion (e.g., by calculating an average value to reduce the effects of noise). The direction of acceleration in three dimensions can also be calculated from such data streams, but they are not shown in Figures 4A, 4B, and 4C. Also not shown are the multi-dimensional IMU gyroscope data streams and the pointing vectors pointing toward the Earth's gravity and / or magnetic attraction.

[0088] Figure 4A shows data acquired during a tap using the index finger of the opposite hand (relative to the hand holding the device) to tap the left display of a handheld device, similar to the action shown in Figure 1A. The acceleration traces for the X axis (40a), the Y axis (40b), and the Z axis (40c) were used to calculate the acceleration magnitude |A| at 41, as shown in Equation 1. While the vertical acceleration scale in Figure 4A is not calibrated in absolute values, the values ​​for the X, Y, and Z dimensions can be compared relative to one another. The time bar at 43a indicates 25 milliseconds.

[0089] The occurrence and time of the tap 42a was identified as a result of the acceleration magnitude 41 exceeding a preset threshold 42b. Such a threshold can account for signal noise as well as the amplitude of the acceleration resulting from the tap relative to device movement other than a tap (e.g., to perform other hand gestures). As discussed above, extrapolating samples from the region where the signal exceeded threshold 42a back to the baseline level can improve the accuracy of identifying the time of the tap. Among other distinctive features in the accelerometer data (e.g., compared to the traces in Figures 4B and 4C), the initial acceleration of the tap on the Y-axis is shown as a sharp increase in the positive direction 42d. As discussed further below, there is also a feature 42c in the Z-axis trace that is believed to be the result of contact release between the finger and the left display of the handheld device approximately 5 milliseconds after the initial tap contact.

[0090] FIG. 4B shows data acquired during a tap using the index finger of the opposite hand to tap the right display of a handheld device. The acceleration traces for the X axis (44a), the Y axis (44b), and the Z axis (44c) were used to calculate the acceleration magnitude |A| 45 according to Equation 1. The vertical scales for all acceleration traces 40a, 40b, 40c, 44a, 44b, and 44c are the same in FIGS. 4A and 4B. In FIG. 4B, the time bar 43b represents 25 milliseconds. The occurrence and time of the tap 46a was determined when the acceleration magnitude 45 exceeded a preset threshold 46b.

[0091] In Figure 4B (in contrast to Figure 4A), the initial acceleration on the Y-axis 44b following a tap at 46a is in the negative Y direction 46d. The initial positive acceleration 42d resulting from a tap on the left display shown in Figure 4A and the initial negative acceleration 46d following a tap on the right display in Figure 4B are likely a result of the left and right displays being located on opposite sides of the Y-axis (see Figure 5).

[0092] Similar to Figure 4A, Figure 4B also shows trace feature 46c in the Z axis approximately 5 milliseconds after the initial tap contact. Some traces collected during other taps (not shown) show trace features in different axes at approximately the same time after taps to magnitudes approaching the amplitude of the initial tap. This trace feature may be the result of contact being released between the finger and the device (e.g., there may be mechanical variations from tap to tap in the force generated during separation).

[0093] Figure 4C shows data acquired during the process of tapping the right-most display of a handheld device against a solid surface (e.g., a desktop), similar to the action shown in Figure 3B. Acceleration traces along the X axis (47a), the Y axis (47b), and the Z axis (47c) were used to calculate the acceleration magnitude |A| at 48 according to Equation 1. The vertical scale of all acceleration data is approximately 60% larger (i.e., the peak amplitude is larger in Figure 4C) compared to Figures 4A and 4B. In Figure 4C, time bar 43c indicates a 25-millisecond duration. The occurrence and time of the tap 49a were identified as a result of the acceleration magnitude 48 exceeding a preset threshold 49b.

[0094] When involving the movement of a handheld device toward a solid surface, the acceleration of the device itself can be seen in the data stream (e.g., at 49d). The peak amplitude resulting from such device movement is typically smaller than that resulting from any form of tap. Furthermore, the peak amplitude (e.g., at 49e) of a tap against a hard surface (e.g., a desktop) is typically larger than that produced using a finger (or other softer surface). A tap of a handheld device against a hard surface can also produce high-frequency acceleration (e.g., reverberation) for a period following the tap (e.g., approximately 6 milliseconds at 49c). When a handheld device is returned to a solid surface, the magnitude of the acceleration trace |A| at 48 does not immediately return to the baseline level (e.g., at 49f) until the device user's retraction motion is fully completed. Such differences (e.g., comparing Figure 4C with Figures 4A and 4B) are examples of features that can be used to distinguish between taps using a finger or other object to strike a handheld device and taps using the device to strike another object.

[0095] FIG. 5 illustrates considerations for assigning a coordinate system that can be used to classify taps when identifying tap locations, particularly during numerical approaches. The coordinate system shown in FIG. 5 is comprised of conventionally labeled X, Y, and Z axes 55a, 55b, and 55c, with origin 54 at the geometric center of handheld device 50. Additionally, rotational motion (i.e., corresponding to pitch, roll, and yaw) about X, Y, and Z axes 56a, 56b, and 56c can also be expressed. Other coordinate systems are possible, including, for example, using polar coordinates and / or locating the origin at the geometric or operational center of an IMU component located within device 50 (not shown). If one or more IMU components have the capability to sense the Earth's gravitational and / or magnetic pull, the coordinate system of handheld device 50 can also be expressed relative to a vector 57b oriented in the direction of the Earth's gravitational pull 57a.

[0096] Tapping at various locations on the handheld device 50 results in various translational and rotational motions that are represented using one or more such coordinate systems (i.e., measured using multi-dimensional accelerometers and gyroscopes, respectively). As shown in Figures 4A, 4B, and 4C, the IMU data stream can represent the rotational and / or translational motions resulting from tapping at various (known) locations on the device 50 (a "forward" mathematical approach). Distinguishing such motions (a "backward" mathematical approach) can be used within an algorithmic strategy (e.g., numerical, neural network-based) to classify the tap location.

[0097] As one example, a tap at 52a on the left sphere 51a may result in a distinct rotational motion in the negative direction 56b about the Y-axis 55b combined with a rotational motion in the positive direction 56a about the X-axis 55a. Meanwhile, a tap at 52b on the center sphere 51b may result in little rotational motion 56b about the Y-axis 55b, but a large torque in the positive direction 56a about the X-axis 55a. In further contrast, a tap at 52c on the right sphere 51c may result in a distinct rotational motion in the positive direction 56b about the Y-axis 55b combined with a rotational motion in the positive direction 56a about the X-axis 55a. As yet another example, a tap in the speaker area of ​​53 on the handheld device 50 may result in little rotational motion about either axis, but measurable translational motion along the negative Z-axis 55c. As another example, a tap on the bottom left side 58 of the device may result in a large rotational movement predominantly and distinctly in the positive direction 56c about the Z axis 55c.

[0098] FIG. 6 is a flowchart outlining exemplary steps for classifying a tap location on a handheld device in response to an image, symbol, and / or illumination pattern generated on the device display. In a first step 60a, a user grasps the handheld device (61a) and taps and views the display 62a, 62b, 62c. In a next step 60b, an initial illumination pattern (e.g., including one dot 62a, two dots 62b, or three dots 62c) is projected by the display 61b for viewing by the device user. IMU data is then acquired (60c) and analyzed (e.g., including calculated movement magnitude) to determine whether the movement exceeds a threshold 61c (i.e., whether a tap has occurred). If the threshold is not exceeded (60d), further IMU data is collected (61d). If the IMU data exceeds a threshold or other criteria used to determine the occurrence of a tap (60d), data surrounding the tap determination time is collected and classification of the tap location is performed (60e).

[0099] 6 illustrates the use of a neural network 61e to perform classification process 60e. In this exemplary approach, IMU data from the region around the time the threshold was exceeded is input to the neural network. The most likely tap location (e.g., the leftmost spherical display 63) is generated as the neural network output, optionally along with other attributes such as a confidence level for the most likely tap location and a measure of the force applied during the tap (e.g., related to the type of tap and / or other tap attributes).

[0100] Additionally, temporal data about the tap, including the time of the tap 61f and the interval 64 from the previous tap, is registered (60f), along with a pattern of activity involving two or more previous taps being identified. Based on the location of the tap, along with other tap attributes such as timing and tap force, a new display pattern can be generated (60g) for the device display 61g. The overall process can then be repeated (60b) using the new display pattern.

[0101] 7 is a flowchart outlining example steps for classifying a tap location on a handheld device in response to an auditory cue or sound presented using the device speaker. The sound may include, for example, one or more words, one or more pronunciations, one or more voices, one or more musical tones, one or more animal sounds, one or more nature sounds, one or more bell sounds, one or more chime sounds, and / or one or more warning sounds.

[0102] In a first step 70a, a user can squeeze (71a) the handheld device to tap and hear an auditory cue from a speaker 72. In a next step 70b, a sound pattern (in this case, a series of musical notes 71b) is played from the speaker. IMU data is then acquired (70c) and analyzed (71c) to determine whether the movement exceeds a threshold (i.e., indicates the occurrence of a tap). If the threshold is not exceeded (70d), the process of collecting IMU data is repeated (71d). If the IMU data exceeds the threshold or meets other criteria used to determine the occurrence of a tap (70d), data around the time of the tap determination is used to perform a tap location classification 70e.

[0103] 7 illustrates performing a classification process 70e using a numerical approach 71e. In this case, a tap (73) on the lower left side of the handset by the index finger of the hand opposite the hand used to grasp the device can be identified, for example, based on a dominant rotation signal in the XZ plane 71e (counterclockwise when viewed facing the top of the device). The most likely tap location is calculated, optionally along with other attributes (e.g., tap attributes), such as a confidence level for the most likely tap location and the force applied during and after the tap.

[0104] Additional temporal data about the tap, including the time of the tap 71f and the interval 74 from the previous tap, is registered (70f), along with a pattern of activity when combined with the previous tap. Based on the newly identified tap location, along with other tap attributes, a new sound pattern 71g can be generated and played (70g). The entire process can then be repeated (70b) using a new auditory stimulus.

[0105] The disclosure of the foregoing examples has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many variations and modifications of the examples described herein will be apparent to those skilled in the art in light of the above disclosure. It will be understood that various components and features described in connection with particular examples can be added, deleted, and / or substituted with other examples, depending on the intended use of the example.

[0106] Additionally, in describing representative examples, the specification may present a method and / or process as having a particular order of steps. However, unless the method or process relies on the particular order of steps described herein, the method or process should not be limited to the particular order of steps described. As one skilled in the art would recognize, other orders of steps are possible. Accordingly, the particular order of steps described herein should not be construed as a limitation on the scope of the claims.

[0107] While the invention is susceptible to various modifications and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It is to be understood that the invention is not limited to the particular forms or methods disclosed, but rather the invention is intended to cover all modifications, equivalents, and alternatives falling within the scope of the appended claims.

Claims

1. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; at least one device display attached to the device body and operably coupled to the device processor; the device processor: generating a first illumination pattern by the at least one device display; acquiring inertial measurement data from the at least one inertial measurement unit; determining a location of an initial tap by a device user on one of the device body and the at least one device display based at least in part on the inertial measurement data; and a device configured to generate a second illumination pattern by the at least one device display based at least in part on the initial tap location;

2. 10. The device of claim 1, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

3. 10. The device of claim 1, A device in which orientation data and magnitude data are calculated from the inertial measurement data.

4. 10. The device of claim 1, The device, wherein the device processor is further configured to determine from the inertial measurement data an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction.

5. 10. The device of claim 1, a device, wherein the initial tap at the initial tap location is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a second hand on the at least one device display, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, tapping the handheld device against a solid object, tapping the at least one device display against a solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display against an additional handheld device display.

6. 10. The device of claim 1, 11. A device, further comprising: a tactile unit that alerts a device user that a first lighting pattern is being generated on the at least one device display.

7. 10. The device of claim 1, The device, wherein the first illumination pattern includes one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

8. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; a speaker within the device body operably coupled to the device processor; the device processor: generating a first sound by the speaker; acquiring inertial measurement data from the at least one inertial measurement unit; Identifying an initial tap location by a device user on the device body based at least in part on the inertial measurement data; and The device is configured to generate a second sound by the speaker based at least in part on the initial tap location.

9. 9. The device of claim 8, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

10. 10. The device of claim 9, 10. A device, wherein the initial sound comprises one or more of: one or more words, one or more pronunciations, one or more voices, one or more musical tones, one or more animal sounds, one or more nature sounds, one or more bell sounds, one or more chime sounds, and one or more alarm sounds.

11. 10. The device of claim 9, a device, wherein the initial tap at the initial tap location is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, and tapping the handheld device against a solid object.

12. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; the device processor: acquiring inertial measurement data from the at least one inertial measurement unit; calculating orientation data and magnitude data from the inertial measurement data; determining a location of an initial tap by a device user on the device body based at least in part on one or both of the direction data and the magnitude data; and The device, wherein the device processor and / or a remotely connected processor are configured to perform an action based at least in part on the initial tap location.

13. 13. The device of claim 12, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

14. 13. The device of claim 12, a device, wherein the initial tap at the initial tap location is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, and tapping the handheld device against a solid object.

15. 13. The device of claim 12, A device, wherein additional tap locations on the device body are identified by additional inertial measurement data obtained from the at least one inertial measurement unit.

16. 13. The device of claim 12, The device further comprising one or both of a Wi-Fi communication module and a Bluetooth communication module configured to communicate between the device processor and a remotely connected processor.

17. 17. The device of claim 16, The device, wherein the action is further based on a location pointed to by a device user on a touch-sensitive screen operably coupled to a remote processor when generating an initial tap at the initial tap location.

18. 13. The device of claim 12, The device, wherein the device processor is further configured to determine from the inertial measurement data an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction.

19. 20. The device of claim 18, wherein the action is further based on one or more of an initial time of occurrence identifying the initial tap location, one or more additional tap locations, additional times of occurrence identifying the one or more additional tap locations, and an orientation of the handheld device.

20. 13. The device of claim 12, The device, wherein the system further comprises one or more of one or more push buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor, each operably coupled to the device processor.

21. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; at least one device display attached to the device body and operably coupled to the device processor; the device processor: generating a first illumination pattern by the at least one device display; acquiring inertial measurement data from the at least one inertial measurement unit when a user taps the device in response to the first illumination pattern; determining a location of an initial tap by a device user on one of the device body and the at least one device display based solely on the inertial measurement data; and a device configured to generate a second illumination pattern by the at least one device display based at least in part on the initial tap location;

22. 22. The device of claim 21, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

23. 22. The device of claim 21, A device in which orientation data and magnitude data are calculated from the inertial measurement data.

24. 22. The device of claim 21, The device, wherein the device processor is further configured to determine from the inertial measurement data an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction.

25. 22. The device of claim 21, a device, wherein the initial tap at the initial tap location is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a second hand on the at least one device display, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, tapping the handheld device against a solid object, tapping the at least one device display against a solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display against an additional handheld device display.

26. 22. The device of claim 21, 11. A device, further comprising: a tactile unit that alerts a device user that a first lighting pattern is being generated on the at least one device display.

27. 22. The device of claim 21, The device, wherein the first illumination pattern includes one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

28. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; a speaker within the device body operably coupled to the device processor; the device processor: generating a first sound by the speaker; acquiring inertial measurement data from the at least one inertial measurement unit when a user taps the device in response to the first sound; determining a location of an initial tap by a device user on the device body based solely on the inertial measurement data; and The device is configured to generate a second sound by the speaker based at least in part on the initial tap location.

29. 29. The device of claim 28, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

30. 30. The device of claim 29, 10. A device, wherein the initial sound comprises one or more of: one or more words, one or more pronunciations, one or more voices, one or more musical tones, one or more animal sounds, one or more nature sounds, one or more bell sounds, one or more chime sounds, and one or more alarm sounds.

31. 30. The device of claim 29, a device, wherein the initial tap at the initial tap location is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, and tapping the handheld device against a solid object.

32. 1. A handheld device for interaction by a device user, comprising: a device body configured to be held in a first hand of a device user; electronic circuitry within the device body including a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; the device processor: acquiring inertial measurement data from the at least one inertial measurement unit; calculating orientation data and magnitude data from the inertial measurement data; When a user generates a first tap on the device, determining a location of an initial tap by the device user on the device body based on one or both of the direction data and the magnitude data; and The device, wherein the device processor and / or a remotely connected processor are configured to perform an action based at least in part on the initial tap location.

33. 33. The device of claim 32, 10. A device, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

34. 33. The device of claim 32, the initial tap is generated by a device user by one of tapping a second hand on the handheld device, tapping a finger of a second hand on the handheld device, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, and tapping the handheld device on a solid object.

35. 33. The device of claim 32, A device, wherein additional tap locations on the device body are identified by additional inertial measurement data obtained from the at least one inertial measurement unit.

36. 33. The device of claim 32, The device further comprising one or both of a Wi-Fi communication module and a Bluetooth communication module configured to communicate between the device processor and a remotely connected processor.

37. 37. The device of claim 36, The device, wherein the action is further based on a location pointed to by a device user on a touch-sensitive screen operably coupled to a remote processor when generating an initial tap at the initial tap location.

38. 33. The device of claim 32, The device, wherein the device processor is further configured to determine from the inertial measurement data an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction.

39. 39. The device of claim 38, the action is further based on one or more of an initial time of occurrence identifying the initial tap location, one or more additional tap locations identified based on one or more additional taps, additional times of occurrence identifying the one or more additional tap locations, and an orientation of the handheld device.

40. 33. The device of claim 32, The device, wherein the system further comprises one or more of one or more push buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor, each operably coupled to the device processor.

41. 1. A method for human interaction using a handheld device, comprising: the handheld device includes a device processor, at least one device display attached to a device body and operably coupled to the device processor, and at least one inertial measurement unit within the device body operably coupled to the device processor, the method comprising: generating an illumination pattern on the at least one device display; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, orientation data and magnitude data from the inertial measurement data; when a human generates a tap on the handheld device, determining, by the device processor, a tap location on the device body based on one or both of the direction data and the magnitude data; and performing, by one or both of the device processor and a remotely connected processor, an action based at least in part on the tap location.

42. 42. The method of claim 41, The method, wherein the illumination pattern includes one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

43. 42. The method of claim 41, the tap is generated by a human being by one of tapping a second hand on the handheld device being held in a first hand, tapping a finger of a second hand on the handheld device, tapping a finger of a second hand on the at least one device display, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, tapping the handheld device against a solid object, tapping the at least one device display against a solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display against an additional handheld device display.

44. 42. The method of claim 41, the device processor is further configured to determine from the inertial measurement data an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction, and the action is further based on the orientation of the handheld device.

45. 42. The method of claim 41, the handheld device further includes a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert a human that an illumination pattern is being generated on the at least one device display.

46. 42. The method of claim 41, the handheld device further includes a device speaker operably coupled to the device processor, the method further comprising playing one or more sounds via the device speaker to alert a person that a lighting pattern is being generated on the at least one device display.

47. 1. A method for human interaction using a handheld device, comprising: the handheld device includes a device processor; a device speaker within a device body operatively coupled to the device processor; and at least one inertial measurement unit within the device body operatively coupled to the device processor; and the method includes: generating one or more sounds on the device speaker; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, orientation data and magnitude data from the inertial measurement data; when a human generates a tap on the handheld device, determining, by the device processor, a tap location on the device body based on one or both of the direction data and the magnitude data; and performing, by one or both of the device processor and a remotely connected processor, an action based at least in part on the tap location.

48. 48. The method of claim 47, the one or more sounds include one or more of: one or more words, one or more pronunciations, one or more voices, one or more musical tones, one or more animal sounds, one or more nature sounds, one or more bell sounds, one or more chime sounds, and one or more alarm sounds.

49. 48. The method of claim 47, the tap is generated by a human being by one of tapping a second hand on the handheld device being held in a first hand, tapping a finger of a second hand on the handheld device, tapping a finger of a second hand on at least one device display, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, tapping the handheld device against a solid object, tapping at least one device display attached to the device body against a solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display against an additional handheld device display.

50. 48. The method of claim 47, the handheld device further includes a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert a human that the one or more sounds are being produced by the device speaker.

51. 1. A method for human interaction using a handheld device, comprising: The handheld device includes a device processor and at least one inertial measurement unit within a device body operatively coupled to the device processor, and the method includes: acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, orientation data and magnitude data from the inertial measurement data; when a human generates an initial tap on the handheld device, determining, by the device processor, a location of the initial tap on the device body based on one or both of the direction data and the magnitude data; and performing, by one or both of the device processor and a remotely connected processor, an action based at least in part on the initial tap location.

52. 52. The method of claim 51 , The method, wherein the at least one inertial measurement unit includes one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

53. 52. The method of claim 51 , the initial tap is generated by a human being by one of tapping a second hand on the handheld device being held in a first hand, tapping a finger of a second hand on the handheld device, tapping a finger of a second hand on at least one device display, tapping a finger of a first hand on the handheld device, tapping a body part on the handheld device, tapping the handheld device against a solid object, tapping at least one device display attached to the device body against a solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display against an additional handheld device display.

54. 52. The method of claim 51 , and further comprising identifying, by the device processor, one or more additional taps at one or more additional tap locations on the device body based on additional inertial measurement data obtained from the at least one inertial measurement unit, wherein the action is further based on the one or more additional tap locations.

55. 55. The method of claim 54, and determining, by the device processor, one or more tap intervals between the initial tap and the one or more additional taps, wherein the action is further based on one or more of a time of first occurrence of the initial tap and the one or more tap intervals.

56. 52. The method of claim 51 , The method further includes determining, by the device processor, from the inertial measurement data, an orientation of the handheld device relative to one or both of the Earth's gravity and the Earth's magnetic attraction, wherein the action is further based on the orientation of the handheld device.

57. 52. The method of claim 51 , the handheld device further includes a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert a human that the inertial measurement data is about to be acquired.

58. 52. The method of claim 51 , the handheld device further comprising one or both of a Wi-Fi communication module operably coupled to the device processor and a Bluetooth communication module operably coupled to the device processor, the communication modules communicating between the device processor and a remotely connected processor.

59. 59. The method of claim 58, the action is further based on a screen location pointed to by a human on a touch-sensitive screen operably coupled to a remote processor when generating the initial tap.

60. 52. The method of claim 51 , the handheld device further includes one or more of one or more push buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor, each operably coupled to the device processor.

Images