Determining tap position on handheld electronic device based on inertial measurements

Abstract

Systems and methods for real-time determination of a tap location on a body of a handheld device using a data stream from an embedded inertial measurement unit (IMU) are described. The strike may be generated by striking the handheld device with an object (e.g., a finger) or may be generated by moving the handheld device in a manner that causes the handheld device to strike another object. Indications of IMU accelerometer, gyroscope, and / or (magnetic attraction and / or gravitational attraction with respect to the earth) orientation measurements are examined to differentiate signal characteristics resulting from knocks at some location from knocks at other locations on the device body. Neural networks and / or numerical methods may be used to perform such classification. Knock position, tap timing, and tap properties, such as magnitude of applied force, device orientation, and motion amplitude and direction during and after the tap, may be used to control or modulate responses within the handheld device and / or actions within the connected device.

Description

[0001] Related application data

[0002] This application claims the benefit of and priority to U.S. Application Serial No. 17 / 874,253, filed on July 26, 2022, the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] The present application generally relates to systems and methods for individuals to perform machine-based interactions using handheld electronic devices. Although the handheld device can be used by anyone, it is particularly suitable for use by young children, utilizing simple interactive movements that do not require precise manual dexterity and / or understanding of complex interactive sequences. The systems and methods herein employ techniques from the fields of computer programming, electronic design, firmware design, inertial measurement units (IMUs), ergonomic structure, device control, human motion control, and human-computer interaction. The systems and methods can provide users, particularly young children, with intuitive machine interfaces to quickly and / or instinctively interact in environments consisting of real and / or virtual objects. Background Art

[0004] In recent years, the world has become increasingly dependent on portable electronic devices that have become more powerful, sophisticated, and useful to a wide range of users. However, while children may quickly adopt some aspects of using electronic products designed for more experienced users, young children may benefit from using interactive electronic devices that are small, lightweight, colorful, fun, informative, ergonomically designed for children (including child safety), and easy to use. The systems and methods disclosed herein take advantage of recent advances in the fields of haptic technology, sound generation using microspeakers, portable displays, and inertial measurement units (sometimes also called inertial motion units).

[0005] In handheld devices, alerts can be generated by haptic units (also known as kinesthetic communication) and / or microspeakers. Haptic units typically use an eccentric (i.e. unbalanced) rotating mass or a piezoelectric actuator to produce vibrations that can be felt. Along similar lines, vibrations in microspeakers are typically generated using conventional (i.e., those associated with larger speakers) electromagnetic moving coils or piezoelectric (so-called buzzer) designs.

[0006] A two-dimensional visual display consists of any number of monochromatic or multicolor addressable light sources or pixels. Displays can range from a single light source (e.g., illuminating a sphere via waveguide transmission) to displays capable of displaying a single digit (e.g., a seven-segment display) or alphanumeric characters (e.g., a five-pixel by eight-pixel array), to high-resolution screens with tens of millions of pixels. Regardless of scale, displays are typically implemented as: 1) a two-dimensional array of light sources (most commonly light-emitting diodes (LEDs)), or 2) two polarized glass plates sandwiching a liquid crystal material (i.e., forming a liquid crystal display or LCD), which responds to an electric current by allowing different wavelengths of light from one or more illumination sources (i.e., a backlight) to pass through.

[0007] An inertial measurement unit (IMU) may contain any or all of the following in combination: 1) linear accelerometers that measure forces generated during movement in up to three axes or dimensions (i.e., governed by Newton's second law of motion) 2) gyroscope-based rotation rate or velocity sensing in up to three axes of rotation, 3) magnetometers that measure magnetic fields (i.e., magnetic dipole moments), including those generated by the Earth, and / or 4) measurements of the Earth's gravitational pull (including gravitational orientation) by measuring forces on internal masses. The accuracy of IMUs varies widely, depending on size, operating range, compensation hardware available to correct measurements (affecting cost), environmental factors including thermal gradients, availability of individual device calibrations, and the (integration) time required to perform measurements.

[0008] Advances in electronics (i.e., hardware), standardized communications protocols, and dedicated frequency allocations within the electromagnetic spectrum have led to the development of a variety of portable devices that are capable of wirelessly communicating with other nearby devices and with large-scale communications systems, including the World Wide Web. Considerations in which protocols (or combination of available protocols) to employ in such portable devices include power consumption, communications range (e.g., from a few centimeters to hundreds of meters and beyond), and available bandwidth.

[0009] Currently, Wi-Fi (e.g., based on the IEEE 802.11 family of standards) and Bluetooth (managed by the Bluetooth Special Interest Group / Bluetooth SIG) are used in many portable devices. Less common and / or older communications protocols in portable devices in home environments include Zigbee, Zwave, IR (infrared), and cellular or mobile phone based networks. In general (i.e., with many exceptions, especially considering new standards), Wi-fi (wireless networking) offers greater range, greater bandwidth, and a more direct path to the Internet than Bluetooth. On the other hand, Bluetooth offers lower power consumption, a shorter operating range (which can be advantageous in some situations), and less complex circuitry to support communications.

[0010] Advances in miniaturization, reduced power consumption, and increased complexity of electronics, including those used in displays, IMUs, and telecommunications, have revolutionized the mobile device industry. Such portable devices have become increasingly sophisticated, allowing users to simultaneously communicate, geolocate, monitor exercise, track health, warn of danger, capture video, perform financial transactions, and the like. Systems and methods that facilitate simple and intuitive interaction with handheld devices, particularly for use by children, may be useful. Summary of the invention

[0011] In view of the above, a system and method are provided herein that describes a lightweight, easy-to-use and intuitive handheld device that may be particularly suitable for machine-based interaction by young children. Although the device may be partially accepted by children as a toy, the computational flexibility embedded in the device may allow the device to be used as a means for machine-based and human interaction (especially involving individuals who may be remotely located), games, embodied learning, emotional support, communication, expression of creativity, and enhancement of imagination. In addition, the portable "fun" handheld device may stimulate physical movement in children (or adults), including kinetic movement and kinesthetic activities.

[0012] Young children may observe older children and adults engaging in a variety of activities over extended periods of time with portable devices, such as cell phones and tablets. However, young children typically need to first develop the fine motor skills (e.g., touching a specific icon on a touch-sensitive display / touch-sensitive display) and intellectual sophistication to navigate a graphical user interface (GUI) on such devices to achieve a desired goal. Even the idea of ​​pressing one or more simple buttons (e.g., that may be painted bright colors) to achieve a desired goal is typically a concept that young children must learn first. However, tapping anywhere on the body of a handheld, lightweight device or tapping the device against a solid surface (particularly in response to visual, tactile, or auditory cues) can provide children with an intuitive way to use handheld devices in mixed, real, and / or virtual environments without requiring extensive instruction or fine motor skills.

[0013] Determining the presence (including timing) of a tap and its location on the handheld device (including any attached components) can be determined by classifying electronic signatures in data collected from at least one inertial measurement unit (IMU) embedded within the handheld device. As described in the Background section above, IMU data can be derived from a sensed combination of translational acceleration, gyroscopic (i.e., rotational) velocity, magnetic forces (i.e., forces including those exerted by the Earth's magnetic field), and / or gravity (i.e., forces generated by the Earth's large mass). In most multi-dimensional (i.e., up to three axes) implementations, the IMU data can be viewed as a time-varying sequence of vectors containing information on the magnitude and direction of sensed acceleration, orientation, and / or force.

[0014] A finger (i.e., any hand) is a convenient and intuitive "tool" for tapping handheld devices. In English, the term "finger" can convey an ambiguous meaning. In most scientific fields, a "finger" refers to any appendage of the hand used for manipulation and perception. However, in some cases (including medicine), thumbs (containing two bones or phalanges) are considered different from fingers (containing three phalanges) due to differences in size, rigid structure, joints, and / or function. In the description herein, the terms "finger" or (interchangeably) "digit" refer to any hand protrusion, including the thumb.

[0015] Along similar lines, the verb "tap" is used herein to describe an action involving one object striking another object with a discernible moment of contact, a sudden change in acceleration, a change in direction, and / or other discernible signals (or signals) that contact has occurred between the objects. Once in contact, either object may retract from the impact, sounds may be produced, and / or either object may be temporarily deformed during the impact. As a noun in English, tap has many meanings, including referring to the sound made when objects are struck together. As used herein, "tap" refers to the overall action or process of striking two objects together. In addition, as described below, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0016] As a result of a knock on the body of the handheld device including any attached components (i.e. forming a solid structure), the knock force may be transmitted to one or more embedded IMUs, where such rigid structure transmits force in the direction of the knock during the impact (i.e. during the knock). According to Newton's second law, such force may cause the handheld device to move in three dimensions (i.e., usually represented as X, Y and Z, see Figure 5 ). Forces may also cause rotational movement of the device (i.e., usually described as pitch, roll, and yaw, see Figure 5), where, for example, the geometric center of the device and / or the functional center of the IMU can be used as a reference (i.e., origin) as a method of describing such motion. During mathematical analysis, it may also be convenient to consider other locations (e.g., the contact point of the hand holding the device, the contact point of the tap) to locate the origin of the coordinate system and / or vectors representing forces, orientations (e.g., relative to the earth's gravitational attraction or attraction), accelerations, velocities, and motions.

[0017] A user may generate a tap by tapping the device with a finger of an opposing hand (i.e., the hand opposite to the hand holding or supporting the device), tapping the handheld device with an opposing hand (e.g., knuckles, palm), tapping the device with a finger of the hand holding the device, tapping the handheld device with any other body part of the user, tapping the device with a solid object such as a pen or stylus, or tapping the handheld device itself against an object or surface (i.e., including a body part such as a knee or wrist; or other objects such as images or text within a page of a book, a desktop, a floor, and the like). A tap may also be generated by tapping a handheld device against another handheld device (e.g., when simulating combat with a sword or saber). For example, the resulting action of a tap between devices may be specifically limited (i.e., by sensing the tap location) to tap a selected display on one device onto a target display of a second handheld device (e.g., during the process of exchanging information between devices and / or device users).

[0018] When a location on a handheld device is struck against a surface, there is no requirement that the surface be absolutely rigid. For example, when the device is struck against a person's knee or the palm of an opposing hand, some temporary deformation may occur during the strike, at least at the skin level. As a further example, when a stuffed or stretchable toy is struck, there may be some flexibility in the surface being struck. Along similar lines, when the pages of a book or magazine are struck, there may be some yielding or reactive movement of the struck pages.

[0019] As described in more detail in the specific embodiments below, the IMU data may be subject to one or more classification processes to identify not only the location of the tap on the handheld device, but also other characteristics of the tap, collectively referred to as the "attributes" of the tap. The attributes of the tap include the mechanical characteristics of the object used to tap, the magnitude and direction of the force applied during the tap, the movement of the handheld device after the tap (including the magnitude and direction), the orientation of the handheld device relative to one or both of the magnetic and gravitational attraction or attraction of the earth during the tap, and the like.

[0020] Such differentiation of sensed acceleration and force under the control of the device user can allow the type and / or attributes of the tap to be classified and subsequently used during control activities (e.g., controlling sound, display, tactile feedback, and / or other actions on the handheld device itself; or controlling virtual objects and / or actions displayed on one or more connected devices). In addition, the timing of the taps, particularly the timing of the taps in relation to the timing of one or more previous taps and / or other events in the device user's environment (including taps on one or more other handheld devices), can be a component of additional control functionality for the handheld device user.

[0021] According to one example, a handheld device is provided for interaction by a device user, the handheld device comprising: a device body configured to be held by a first hand of the device user; electronic circuitry within the device body, comprising a device processor; at least one inertial measurement unit within the device body, operably coupled to the device processor; and at least one device display attached to the device body, operably coupled to the device processor, wherein the device processor is configured to: generate a first lighting pattern through at least one device display; obtain inertial measurement data from at least one inertial measurement unit; determine an initial tap location of the device user on one of the device body and at least one device display based at least in part on the inertial measurement data; and generate a second lighting pattern through at least one device display based at least in part on the initial tap location.

[0022] According to another example, a handheld device for device user interaction is provided, the handheld device comprising: a device body configured to be held by a first hand of a device user; an electronic circuit within the device body, comprising a device processor; at least one inertial measurement unit within the device body, which is operably coupled to the device processor; and a speaker within the device body, which is operably coupled to the device processor, wherein the device processor is configured to: generate a first sound through the speaker; obtain inertial measurement data from the at least one inertial measurement unit; determine an initial tap position of the device user on the device body based at least in part on the inertial measurement data; and generate a second sound by the speaker based at least in part on the initial tap position.

[0023] According to yet another example, a handheld device for device user interaction is provided, the handheld device comprising: a device body configured to be held by a first hand of a device user; an electronic circuit within the device body, comprising a device processor; at least one inertial measurement unit within the device body, which is operably coupled to the device processor, wherein the device processor is configured to: acquire inertial measurement data from the at least one inertial measurement unit; calculate direction data and amplitude data based on the inertial measurement data; determine an initial tap location of the device user on the device body based at least in part on one or both of the direction data and the amplitude data; and perform an action based at least in part on the initial tap location on one or both of the device processor and a remotely connected processor.

[0024] Other aspects and features, including the need and use of the present invention, will become apparent from the following description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] A more complete understanding may be derived by referring to the detailed description when considered in conjunction with the following illustrative drawings. In the drawings, like reference numerals refer to like elements or acts throughout the drawings. The presented examples are illustrated in the drawings, in which:

[0026] Figure 1A Tapping the leftmost device display (projecting the letter "C") using the index finger of the hand opposite the hand holding the handheld device is shown as a means of interaction for the device user.

[0027] Figure 1B yes Figure 1A Continuation of the illustrated scenario, where the index finger of the hand opposite the hand used to hold the handheld device taps the central device display (projecting the letter "A") as a means of machine-based interaction.

[0028] Figure 2 Using the thumb of the same hand used to hold the handheld device (eg, in response to interactive sounds produced by the speaker) to tap the upper body of the handheld device including in the area containing the device speaker is shown as a means of interaction.

[0029] Figure 3A Tapping a central display (projecting an image of an aquatic animal) of a rigid object (such as the surface of a table or desk) with a handheld device is shown as a means of interaction for the user of the device.

[0030] Figure 3B Similar to Figure 3A The scene shown shows tapping the rightmost display (showing projected symbols representing speech) on a rigid object (such as the surface of a table or desk) as a means of human-computer interaction.

[0031] Figure 4Ashows the use of a finger to tap the left display of the handheld device (see Figure 1A ), illustrating a method that can be used to detect the occurrence and location of a tap.

[0032] Figure 4B 1 shows exemplary acceleration data and calculated acceleration magnitude traces during the time when the right display of a handheld device is tapped by a finger, showing (i.e., compared to the traces showing when the left display is tapped) Figure 4A The acceleration difference (compared) can be identified to help determine the tap location.

[0033] Figure 4C 1 shows exemplary acceleration data and calculated acceleration magnitude traces during the time when the right display of a handheld device strikes a tabletop, illustrating the difference between when the display strikes a solid surface and when the right display is tapped with a finger (e.g., Figure 4B The difference in the trajectory when compared to the trajectory shown in ) (i.e., can be used to identify the tapping pattern or attributes).

[0034] Figure 5 Examples of coordinate systems and rotation axes are shown that may be used to describe movement, velocity, and acceleration in different dimensions as part of the process of calculating the location of a tap on a handheld device from acquired IMU data.

[0035] Figure 6 is a flow chart outlining exemplary steps for processing and locating tap-based interactions after projecting an illumination pattern on one or more handheld device displays.

[0036] Figure 7 is a flow chart outlining exemplary steps for processing and locating a tap-based interaction following a sound produced by a handheld device speaker or buzzer. DETAILED DESCRIPTION

[0037] Before describing the examples, it should be understood that the present invention is not limited to the specific examples described herein, which, of course, can be varied. It should also be understood that the terms used herein are only used to describe specific examples and are not intended to be limiting, because the scope of the present invention will only be limited by the appended claims.

[0038] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. It must be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, reference to "a compound" includes a plurality of such compounds, reference to "the polymer" includes reference to one or more polymers and equivalents thereof known to one of ordinary skill in the art, and so on.

[0039] Where a range of values ​​is provided, it is understood that, unless the context clearly specifies otherwise, each intermediate value (to one tenth of the unit of the lower limit) between the upper and lower limits of the range is also specifically disclosed. Each smaller range between any of the values ​​or intermediate values ​​in the range and any other values ​​or intermediate values ​​in the range is included in the present invention. The upper and lower limits of these smaller ranges may be independently included in or excluded from the range, and each range in which one, neither, or both limits are included in the smaller range is also included in the present invention, subject to any specifically excluded limitations in the range. Where the range includes one or both limits, the present invention also includes ranges that do not include one or both of these included limits.

[0040] Certain ranges are given herein, and numerical values ​​are preceded by the term "about." The term "about" is used herein to provide literal support for the exact number preceding it, as well as numbers that are close to or approximately that number. In determining whether a number is close to or approximately a specifically cited number, a close or approximately unquoted number may be a number that, in the context in which it is presented, provides a substantial equivalence to the specifically cited number.

[0041] As described above in the Summary of the Invention, a strike can be generated by intentionally moving an object and then causing the object (i.e., a "strike object") to strike a location on the surface of a handheld device selected and / or aimed by a user (i.e., a "strike location"). For example, the strike object can be a finger of an opposing hand, any other body part, a finger of a hand holding the device, a stylus, a stick, a pencil, and the like.

[0042] A knock force may also be generated by colliding (i.e., intentionally moving) a target location or area on the surface of the handheld device against a solid surface (or an at least partially rigid surface such as human skin, a stuffed toy, or the pages of a book). In combination with these motions, the handheld device and another object (e.g., a child's toy, a hand) may be simultaneously moved toward each other. In general, the relative movement of the handheld device and the surface being knocked determines the characteristics of the knock (e.g., peak force, acceleration, calculated knock location) and which object was moved (e.g., relative to the ground or other objects in the user's environment). IMU data streams before and after the knock may help determine whether a stationary device was knocked with an impact object, whether the device was forcibly moved toward another object (e.g., see Figure 4C ), or whether these two processes occur simultaneously.

[0043] As mentioned in the background section above, an IMU data stream may consist of one or more of the following:

[0044] 1. Up to three channels (i.e., representing three spatial dimensions, usually X, Y, and Z; see Figure 5 ), and

[0045] 2. Up to three channels of gyroscope rotation speed (i.e., representing rotation around 3 axes; see Figure 5 ).

[0046] Both acceleration and gyroscope data can be expressed as one or more time-varying vectors (see ) using, for example, Cartesian, polar, and / or spherical coordinate systems. Figure 5 ).

[0047] Optionally, a data stream indicating the orientation of the handheld device relative to one or both of the gravitational and magnetic forces of the earth can be considered. Such orientation data can be an integral part of the process of determining a tap location. For example, if a device user (i.e., in any of up to three spatial dimensions) holds the device "upside down" (or at least not in the expected manner) in his hand, a transformed coordinate system reflecting a measured or atypical device orientation can be used to account for the expected acceleration and / or gyroscope data streams. In addition, potential enumerated tap locations can be redefined based on the possibility that one or more surfaces of the handheld device may be more accessible while other surfaces may be hidden due to such repositioning of the hand holding the device.

[0048] Device orientation data may also be used to represent movement of a handheld device relative to previous movement of the device (e.g., when sequential rotation of the device is sensed to be similar to turning a conventional knob), and / or relative to movement of other objects in the device user's environment (such as a visual display screen). In an additional implementation example, if the orientation of a vertically oriented display screen is known (e.g., relative to gravitational attraction), and the IMU data stream includes the direction of gravitational attraction relative to the body of the handheld device, a displayed image of movement of the handheld device (or any virtual object controlled by the device, such as a puppet or virtual toy) may be made to appear in the same orientation as the physical device, or at any other selected viewing angle. In a further example, the orientation of the device (i.e., relative to the magnetic or gravitational attraction of the earth) during a tap may be used to define one or more tap attributes (described more fully below), which may be used as an integral part of controlling functionality of the handheld device by controlling or modulating actions on the device itself and / or a connected device (e.g., a connected tablet).

[0049] Determining the location of a tap on a handheld device can be viewed as a classification process that uses patterns or "signatures" within the IMU data stream (e.g., compared to a template) to calculate the most likely tap location. The initial step in this classification process is to identify one or more methods to specify potential tap locations on the surface of the handheld device. One method of indicating potential tap locations involves coating the surface of the device with a virtual grid (e.g., composed of triangles, finite elements) or a rectangular grid pattern (i.e., a rectangular grid pattern in a curved two-dimensional or three-dimensional space), where the mesh or grid intersections can represent potential tap locations. This process effectively assigns a uniform spatial distribution of potential tap locations to the surface of the handheld device. One disadvantage of this approach is that taps at two or more nearby grid locations (i.e., at least within the variability of different users, different tap intensities, different handles, etc.) may not produce tap signatures that can be distinguished from each other (i.e., uniquely classified).

[0050] Another exemplary method of identifying tap locations involves enumerating a desired number of distinct locations on the surface of the device. In effect, enumeration allows for the folding of a multi-dimensional array of locations on the device into a single targeted classification data set, where the number of potential tap locations is related to (i.e., effectively bounded by) the spatial resolution of the tap locations within a region. Such an enumeration scheme allows the density of potential tap locations to vary across the surface of the device, for example, when determining tap locations in some areas of the device (e.g., in areas typically covered by a hand) and / or when using applications that may benefit from more closely spaced tap locations (e.g., in areas surrounding one or more displays, see Figure 1A ), taking into account the lower spatial resolution.

[0051] As an example during different applications, only three tap locations may be required in response to a simple query (e.g., on each of the three displays, see Figure 1A and Figure 1B ); while other applications may utilize or attempt to discern dozens of different tap locations on various surfaces of the device. In addition to the tap location on one or more attached displays (see, for example, Figure 1A and Figure 1B ), the tapping location may also include the area of ​​the speaker or buzzer (see Figure 2 ), anywhere on the top surface of the device body, an edge or corner of the device, or any surface on the opposite side or edge of the device body, or the external elements just listed.

[0052] Analog IMU data is converted to a digital form suitable for numerical processing, and analog-to-digital (A / D) conversion techniques well known in the art can be used. IMU sampling rates are typically in the range of about 100 samples / second to about 10,000 samples / second, where (as further described in the background section above) higher IMU sampling rates involve trade-offs in signal noise, cost, power consumption, and / or circuit complexity. Because a tap is a relatively fast event (e.g., typically lasting in the range of about 10 to 100 milliseconds), a higher IMU A / D sampling rate may allow for identification of a more distinct tap location on a handheld device.

[0053] In an example of an implementation, the IMU sampling rate can be changed dynamically. For example, a slower sampling rate can be used most of the time to save power and computing resources (e.g., when identifying non-tap, hand movement-based gestures). Once an increased signal is sensed in the accelerometer and / or gyroscope data stream (i.e., indicative of a tap), the A / D sampling rate can be increased to the maximum value available to the IMU device. Alternatively or in addition, the high-rate samples can be buffered via hardware or firmware (e.g., stored in a circular buffer). When a potential tap is detected, one or more processors can retrieve samples in the buffer before the tap is detected to more accurately assess, for example, the earliest time of the tap (e.g., extrapolating one or more signals that exceed a threshold back to a baseline) and include additional samples for classifying the acquired data into a tap location. In a further alternative or additional example, the hardware can perform threshold detection (see Figure 4A , Figure 4B and Figure 4C ) interrupts execution (i.e., execution of firmware or software) within one or more processing elements whenever a hardware element detects a tap.

[0054] Broadly speaking, the classification process of converting IMU data into tap locations may include two overall approaches: 1) numerical methods, which may include multi-channel template matching and / or frequency domain (e.g., Fourier transform) analysis, and 2) neural network methods, in which the network is trained using a dataset of taps of known locations collected under various conditions (e.g., different users, different fingers or other objects used for tapping, various hand sizes and / or orientations holding the device, different tapping forces, different device orientations). Such a training dataset can be obtained from a single user (or a small number of users) who may have a limited force range and / or tap location range (e.g., due to age or articulatory ability). Alternatively or in addition, a wide range of users can be used to generate a neural network training dataset (e.g., particularly involving taps using fingers), which can then be applied globally (i.e., to any device user).

[0055] Numerical methods can utilize a computational model of a handheld device that is coupled to the contact position of the hand holding the device. Based on these considerations, the expected amplitude, timing, and direction of the force applied during a tap at different locations on the device can be estimated. Then, template matching techniques can be applied to all or a subset of the IMU data stream. For example, a template waveform (e.g., modeled and / or empirically acquired) from each potential tap location can be compared with the acquired data to calculate a correlation coefficient (or similar measurement). The template waveform associated with the position with the highest correlation corresponds to the most likely tap orientation. Alternatively or in addition, simply comparing the sequence of positive and negative peaks in the acquired data with the sequence of positive and negative peaks in the template waveform (e.g., on all available IMU channels) may be sufficient to uniquely identify the tap location, particularly when there is a small number of target tap locations (e.g., based on the interactive activity being performed).

[0056] During such calculations, it may be helpful to use one or more polar and / or spherical coordinate systems to more easily account for, for example, three-dimensional (e.g., rotational) velocity and acceleration vectors about a point of contact including the IMU functional center and / or the hand holding the device. It may also be helpful to convert the time-based IMU data stream to the frequency domain (e.g., processed using a multidimensional Fourier transform) to determine the presence of characteristic frequency components and phase differences between frequency components (i.e., time and / or sequence related) within the data stream, which may help distinguish different tap locations.

[0057] The neural network method can include using the IMU data stream as an input to one or more neural networks, the neural network including, for example, a single output identifying the maximum likelihood tap location (e.g., indicating an enumerated tap location) or a binary (i.e., yes / no) output associated with each location. Either neural network configuration can be additionally configured to provide a confidence level (or equivalent measure) associated with the degree of match for each potential tap location. The neural network can also be trained to output one or more tap attributes described in more detail below. The tap attributes include, for example, the force of the tap, the orientation of the handheld device, the movement of the device when tapping, whether it was hit by a soft or hard object, whether it was hit by a finger of an opposing hand (relative to the hand holding the device), and the like.

[0058] Digital signal processing (DSP) techniques known in the art for such time-varying data streams can be used as follows: 1) Multilayer Perceptrons (MLPs) and 2) Recurrent Neural Networks (RNNs) are examples of network topologies (i.e., suitable for processing time-based data streams). Backpropagation can generally be used to train such networks. Optionally, the neural network can be made adaptive by recording when corrections are made in the virtual activity controlled by the handheld device. Such adaptive corrections can further train the neural network using supervised learning methods.

[0059] The algorithm that converts IMU data into tap locations can target any number of different tap locations, depending on factors such as the IMU sampling rate, noise in the IMU data, the mechanical design of the handheld device, the computational approach, and the temporal / mechanical consistency of the taps generated by the user (or other users of the device). The tap locations on the handheld device can be used as input to control or modulate resulting "actions" in the real and / or virtual world. Such actions may be: 1) strictly limited to the handheld device, such as tactile vibrations, sounds, or symbols displayed on the handheld device, 2) transmitted to another device in the device user's environment (e.g., sounds produced by nearby speakers or images displayed on nearby screens), and / or 3) transmitted to a remote device (i.e., handheld or otherwise), where the taps and / or their locations may signal a display, audio, and / or other indicator to the remotely connected individual.

[0060] As further described below, the time at which a tap occurs (including the interval between taps) can additionally be used as a component of controlling or modulating the post-tap action. A variety of methods can be used to determine the time at which a tap occurs, including:

[0061] 1. The time at which an acceleration or velocity component greater than a threshold level is initially detected, optionally including a technique for extrapolating multiple samples collected before the threshold was exceeded back to a baseline as a means of improving the accuracy of the estimate of the onset time,

[0062] 2. The time of maximum (i.e. peak) acceleration or velocity, where multiple sampling points may also be considered (e.g., fitting a parabola) to improve the accuracy of determining such peak, and / or

[0063] 3. Midway between initially exceeding the threshold level and recovering below the threshold (or any other chosen reference), where extrapolation techniques (e.g., linear fit) taking into account multiple samples within the threshold region may be used to improve accuracy.

[0064] In addition to using the time between taps as an input to control or modulate the resulting action, the time between a "stimulus" (such as a tactile vibration, sound, or symbol displayed on a handheld device) or other stimulus sensed in the device user's environment (e.g., a broadcast sound, an image displayed on a remote screen) and the time of the response tap can also optionally be used as an additional modulator of the action. Such tap timing commands can be intentional (i.e., the device user intentionally times the time when one or more tap responses occur) or unforced (i.e., the device measures the stimulus response time in a manner unknown to the user). The resulting action controlled or modulated at least in part by the stimulus-response tap timing can be within the handheld device (e.g., displaying one or more symbols, generating one or more sounds, tactile vibrations) and / or the tap timing can be transmitted to other devices (e.g., tablets, laptops, e-books), where the resulting action is performed or modulated at least in part by measuring the tap timing generated on the handheld device.

[0065] In another exemplary configuration, the time between taps can be measured using two or more handheld devices (e.g., by two or more juxtaposed individuals or individuals who may be remotely located and connected via telecommunications). For example, a device user can initiate an action in a virtual environment by tapping a first handheld device. Within a predetermined time, a second user can complete a virtual action by tapping a second device. Measuring time between taps and / or other control functions using two or more handheld devices can be performed by any number of device users. This multi-user shared control using taps may be particularly useful during gaming activities and other shared activities in a virtual environment. Shared control of such activities (including the use of handheld devices) is described in U.S. Patent No. 11,334,178, filed on August 6, 2021, and Application Serial No. 17 / 531,571, filed on November 19, 2021, the entire disclosure of which is expressly incorporated herein by reference.

[0066] Optionally, in examples of additional implementations, device orientation can be used as an input to control or modulate a resulting action. The device orientation can be determined based on an IMU component that senses the direction of the Earth's gravitational pull and / or magnetic field, as described in more detail above. For example, a tap performed when the handheld device is held substantially horizontally can be used to indicate a "no" answer to the device user during a conversation; however, when the handheld device is held vertically, a "yes" answer may be indicated. The orientation of the handheld device can also be used within a continuous range of orientations during one or more taps (e.g., partially up or down, similar to the orientation of hands on a clock). For example, the handheld device orientation can be used to select a particular hue or color when drawing (i.e., within a continuous range).

[0067] In a further example, along similar lines, the force of the knock (e.g., determined in particular from the magnitude of the amplitude peak within the IMU data stream) (e.g., see Figure 4A , Figure 4B and Figure 4C ) can be used to convey a continuous range of control or modulation of the resulting action. For example, when the orientation of the device is used to indicate a "yes" or "no" answer, as previously described, a firm tap can be used to indicate a high degree of certainty on the part of the device user that the response is correct. Conversely, a light tap can be used to indicate uncertainty in a response.

[0068] In addition to peak force, other waveform features or "signatures" in the IMU data stream can be used to identify the source of the tap, which in turn can be used to control or modulate the resulting action. Such signatures of the IMU data might include tap direction, duration, frequency components, and phase differences, which might, for example, help identify the hand or specific finger used to produce the tap. At least among a minority of device users, the tap signatures can even distinguish which user is holding the device based on factors such as hand size and grip of the device (i.e., determining the resulting device movement caused by the applied force during the tap) and / or how the device is held in a larger hand versus a smaller hand (i.e., resulting in different contact points that restrict device movement during the tap). In a further example, the tap signature can distinguish between tapping the device with a stylus or pencil (vs. a finger).

[0069] In a still further example, tapping a device against a rigid, non-movable surface (such as a table or floor) may produce a different response than tapping a more forgiving surface (such as a page in a book or stuffed toy). Compared to tapping soft tissue in most superficial body parts, tapping a rigid structure will generally produce a tap force (i.e., as sensed by one or more IMUs) that is shorter in duration, higher in peak amplitude, and contains higher compression wave frequency components throughout the tap (see Figure 4C). In contrast, striking an object that may provide some "give" during the strike will typically produce a force and resulting (i.e. measured) acceleration that is longer in duration, smaller in peak amplitude, and contains a lower compression wave frequency.

[0070] The physical position of the handheld device and its orientation in (three-dimensional) space can be further determined by camera-based tracking of the handheld controller. Camera-based measurements of the handheld controller (e.g., orientation in the camera's field of view, including orientation, position, velocity, acceleration relative to other objects in the field of view) can be combined with the IMU-based data stream to provide further control or modulation of the resulting action. Systems and methods for determining such camera-based measurements are described in U.S. Patent No. 11,334,178, filed on August 6, 2021, and application serial No. 17 / 531,571, filed on November 19, 2021, the entire disclosures of which are expressly incorporated herein by reference.

[0071] In summary, the identification of a tap and its location on a handheld device may be viewed as only one of many control characteristics (e.g., "tap attributes") that can be used to direct or modulate (i.e., influence) an action. The following list summarizes the measured attributes of a handheld device tap, which in turn can facilitate interaction with actions and / or activities embedded in the handheld device and / or can be transmitted to other electronic devices (e.g., a tablet, laptop, e-book, or magazine). The tap attributes can be used to interact with the real world (e.g., control IoT components such as light switches and thermostats) and / or interact in a virtual environment (e.g., control a cartoon avatar and turn pages in a virtual book):

[0072] 1. The presence of a tap and the timing of its occurrence, particularly in relation to the occurrence of other events in the device user's environment, can be used as a primary control feature. As a visual example, a tap (i.e., anywhere on a handheld device) can be used to control the timing of page turns in a virtual book displayed on a connected tablet device. As an auditory example, the presence of a tap can be used to indicate a selected answer immediately following the pronunciation of an answer from a series of possible answers broadcast using a speaker (e.g., on the handheld device or a nearby electronic device).

[0073] 2. The determined tap location on the handheld device can be used to control or modulate actions, particularly when making so-called "N-choose-1" selections. For example, the surface of the handheld device can be "mapped" to the appearance of a human shape, avatar, animal, or other object displayed on a connected screen. In this example, the top, bottom, and sides of the handheld device can correspond to the top, bottom, and sides of the object appearing on the screen (i.e., similar spatial alignment). Tapping at a specific location on the handheld device can be used to indicate that subsequent movement of the handheld device is reflected in the movement of the component selected by the tap (e.g., the head or arm of a puppet).

[0074] 3. The applied force or intensity used to tap can be measured (e.g., in the peak amplitude of acceleration). Light taps can be distinguished from more forceful taps, for example, indicating confidence in an answer or urgency to take action under control of a handheld device.

[0075] 4. In addition to the overall force applied during a tap, the (more subtle) timing of different force signatures can also be used to identify or discern characteristics of the object that impacted the handheld device. For example, a tap with a pencil or stylus can be distinguished from the use of a finger of the opposing hand (e.g., a relatively soft finger). Slight movement prior to impact can distinguish between the use of a finger on the opposing hand versus a finger of the hand holding the device. Considering the movement of the device over a longer period of time may help distinguish whether it is a moving object impacting the device or the moving handheld device itself impacting another surface.

[0076] 5. The interval between any two taps can often be precisely controlled by the device user, and subsequently measured. As a simple example, distinguishing between long duration and short taps provides a simple means for binary (e.g., "yes" and "no") controls. Measuring the continuity of such intervals (i.e., at least up to the ability of a typical user to accurately indicate the time interval) can be particularly useful. For example, the interval between taps can be used to control screen brightness (i.e., over a continuous range of screen illumination).

[0077] 6. Determining multiple time alignments and / or tap patterns provides a wider range of time control capabilities. Tap intervals can be compared to a time standard (e.g., one second, two seconds, etc.) or to each other (e.g., a short tap followed by two longer taps) to encode the meaning of the tap sequence. Greatly expanding the concept of conveying control through tap patterns, taps can be used to indicate musical melodies, pre-specified interpretations (e.g., long intervals to increase hours, short taps to increase minutes, and setting the clock at the same time), and even Morse code.

[0078] 7. The movement of the handheld device between (and during) taps can provide further attributes during the control functions enabled by the device. The amplitude and / or direction (including speed and / or acceleration) of the movement between taps and tap intervals can be considered. As a simple example, the direction of movement before or after a tap can be used to indicate whether the page turns in a virtual book should be in a forward direction or in the opposite direction. In a further example that takes into account both the amplitude and direction measured between taps, if an object on the screen is the focus of attention (e.g., a person speaking), an object displayed nearby can be designated as a new focus by the handheld device user by indicating the distance and direction between the two taps (i.e., relative to the previous focus position).

[0079] 8. The orientation of one or both of the gravitational and magnetic attraction or attraction of the earth during the tap can provide additional tap attributes. As an example of a binary choice, tapping the handheld controller when the handheld controller is raised vertically can be used to indicate approval of the selection (e.g., indicated by another person, visualized on the screen, or uttered through a speaker), while tapping can indicate disapproval when the device is held substantially horizontally. As a further example involving the use of device orientation for a continuous range of controls, the handheld device can be held vertically and then rotated and tapped (e.g., while visualizing the face of a traditional clock) to control the intensity of a connected light (e.g., via the Internet of Things). If the orientation of one or more other objects is known (e.g., relative to the earth's gravitational attraction), the orientation of the handheld device relative to these objects can be calculated. In other words, the earth's gravitational attraction can provide a common reference direction for the handheld device and other objects. For example, if the display screen is oriented vertically, the three-dimensional orientation of the handheld device when tapped can be used to calculate the three-dimensional viewing angle of one or more items drawn on the screen (e.g., including a puppet or an image of the handheld device itself).

[0080] 9. Stroke attributes generated on one handheld device can be combined (i.e., in real time) with stroke attributes generated on one or more additional handheld devices. Additional devices can be collocated with the device user and / or used by a remotely located interconnected user. A second handheld device can even be controlled by a second hand of the device user.

[0081] Considering these handheld device operations as a whole, multiple tap attributes can be used in various combinations to intuitively and efficiently generate a large number of real-time controls in a virtual environment, which in turn can also control various aspects of the real-world environment. As an example of the latter, the controller can be used to tune a multi-speaker sound system, where, once a speaker volume control function is selected, the tap location on the handheld device is used to specify which speaker or speaker combination to be controlled. Multiple speakers can be specified by quickly tapping two or more times at different locations on the handheld device. The device can then be rotated within the front plane to control the speaker volume (one or more) (e.g., similar to a volume control knob), as well as moved in and out of that plane to control the balance of bass and treble.

[0082] As a further example, in this case, in a virtual environment, and using simple, intuitive controls that may be suitable for young children, a handheld device can be used to control a range of aspects of a puppet (or other character, avatar, or object) displayed on one or more connected display devices (this may also include remotely connected display devices that others can see). A single tap on the body of the handheld device can specify that the (rotational and / or translational) movement of the device can be reflected in the overall body movement of the puppet. A single tap on the side of the handheld device may indicate a desire to control the movement of a specific limb. Multiple, closely timed taps can be used to indicate a desire to control multiple limbs simultaneously via movement of the handheld device.

[0083] As described in more detail above, the spatial or "positional resolution" (i.e., the number of different positions that can be determined by the classification process) can vary in different applications, for example, from just a few tap locations (e.g., responding to a yes / no question) to dozens or more possible different tap locations (e.g., identifying different anatomical locations and costumes or actions to be controlled on a puppet). The number of potential locations can even vary from tap to tap. For example, a response to a series of questions in a quiz can be selected from one of three possible answers (i.e., each answer is indicated by a different tap location) to solve the initial question, while the next question in the quiz can be selected from six possible answers (i.e., again, each potential answer is associated with a different tap location).

[0084] In other examples of implementations, the problem of classifying a dynamically changing number of locations can be solved by using different classification processes (e.g., different neural networks trained based on knocks at different numbers of potential knock locations) or by combining the outputs of the classification processes to "lump together" two or more classified locations into one result. The latter approach can also aggregate or combine confidence levels (or other statistical measures) to determine a most likely cumulative knock location or region. For example, these confidence levels can be summed or accumulated to obtain a summary or regional confidence level. In some unusual cases using this approach, the cumulative probability of a set of two or more classified locations with moderate certainty may exceed a single location with a higher level of certainty but surrounded by classified locations with a lower level of certainty.

[0085] In a further example, use of a tap location and / or signaled motion (e.g., following a tap) by a device for controlling a virtual action or object can take into account the interaction context, resulting in the concept of "interpretive control." "Interpretive control" can relax constraints on the tap location(s), orientation(s), and / or motion(s) specified by one or more controllers based on context. For example, if a response to a visual or auditory query is intended to be based on tapping one of three displays, and the correct response is indicated by tapping the left display of the handheld device (e.g., see Figure 1A ), then interpretive controls can allow taps anywhere on the left side of the handheld device (e.g., particularly by young children) to be interpreted as the correct answer.

[0086] Interpretive control can reduce the tap location precision, timing constraints, repetition rate, and / or number of degrees of freedom required to control a device by making assumptions about user intent based on the interaction context. Examples of contextual interpretation that may result in a relaxation of controller precision include specifying a choice from a limited number of viable choices, using one or more previous and / or frequent choices to rank choices at more easily accessible tap locations (e.g., Figure 1A and Figure 1B tapping on the display as shown), using tap patterns that make similar selections easy to indicate (e.g., double-clicking at one location indicates that a previous tap pattern at the selected location should be repeated), and so on.

[0087] Interpretive control can be applied not only to enact or modulate virtual activities based on the location of the strike, but also based on properties of the strike, such as device orientation and / or movement during or after the strike. For example, device movement during and / or after the strike can be used to hammer a virtual nail (e.g., displayed on a connected display device). Interpretive control can allow any device movement after the strike (e.g., in any direction) to produce a video sequence of the nail being hammered (e.g., without requiring directional impact on the nail head or any minimum movement speed of the nail moving within the virtual board). In addition, using interpretive control, the nail can be fully driven into the virtual board after only a limited number of movements (e.g., three) representing the hammering. Along similar lines, rotation of the handheld device after the strike can be used to control a virtual screwdriver, where any degree of rotation can result in the screw being fully turned, and (similar to the nail hammering just described) a limited number of rotations can fully insert the screw.

[0088] Interpretive controls may be particularly useful in interactions involving very young people, elderly people, or people with reduced motor and / or cognitive function. Additional aspects of "interpretive controls" are more fully described in U.S. Patent No. 11,334,178, filed on August 6, 2021, and co-pending application Ser. No. 17 / 531,571, filed on November 19, 2021, the entire disclosures of which are expressly incorporated herein by reference. Determining context from audiovisual content and subsequently generating interpretive controls based on such context is more fully described in U.S. Patent No. 11,366,997, filed on April 17, 2021, the entire disclosure of which is expressly incorporated herein by reference.

[0089] In a further example, "two-handed control" of one or more virtual objects can be achieved by combining the ability to specify a location on a touch-sensitive display using one or more fingers of one hand (or using one or more pointing instruments, such as a stylus) with the ability to generate additional activity control and / or modulation functions substantially simultaneously using the other hand via a handheld device. Two-handed control that combines a determined location on the touch-sensitive display with measured characteristics of taps on the handheld device (e.g., tap location, timing, and / or attributes) can produce multiple degrees of freedom in controlling virtual objects or activities. In the case of a single user, touching the touch-sensitive display / touch-sensitive display with the fingers of one hand may limit the ability to tap the handheld device to using the fingers of the hand holding the device and / or tapping the device on a solid surface. U.S. Patent No. 11,334,178, filed on August 6, 2021, more fully describes various aspects of two-handed control for a single device user, the entire disclosure of which is expressly incorporated herein by reference.

[0090] Dual manual control can also be achieved by one user specifying a location on a touch-sensitive screen / touch-sensitive screen, while substantially simultaneously, a second user's finger (or other tapping mechanism) can generate taps on the handheld device. Such control functionality by two independent users (possibly interacting remotely via telecommunications) can collaboratively perform and / or modulate virtual actions and activities. Two-handed collaborative control by two device users is further described in co-pending application serial number 17 / 531,571, filed on November 19, 2021, the entire disclosure of which is expressly incorporated herein by reference.

[0091] In other examples, while not strictly "handheld," portable electronic devices may be attached and / or manipulated by other parts of the human body. For example, a device in which tap locations are determined based on an IMU data stream may be attached to an arm, leg, foot, or head. Such positioning may be used to address accessibility issues for people with limited upper limb and / or hand motion, people without hands, and / or in other situations where hands may be required for activities. Tap timing and location may be tracked based on tapping motions maintained or generated by other body parts.

[0092] The handheld device may additionally include one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, one or more cameras, an optical cardiac sensor, and an electrical cardiac sensor, each operably coupled to the device processor. Combined with tap location, timing, and attributes, these components can provide additional means for the user to control actions using the handheld device. In addition, a battery powering the electronic components can allow the handheld device to operate without any other power constraints.

[0093] In a further example, sensing of tap locations based on an IMU data stream can largely (although not necessarily) eliminate the need for buttons and / or other forms of (e.g., finger-based) contact controls; however, the elimination of physical switches does not mean that the appearance of contact structures is also eliminated. In fact, images of buttons or any other symbols can be applied to (e.g., painted on) the surface of the handheld device at different potential tap locations. For example, to cater specifically to young children, brightly colored circles (or other shapes) on the surface of the handheld device can indicate different tap locations. Optionally, these symbols can be applied using temporary adhesives (i.e., including so-called "stickers") that can be exchanged for different user applications, or just for fun as the child grows older and / or develops different personal preferences. Such adhesives and / or fastening components can be elements of different skins and / or other accessories that are used to "dress up" the handheld device (e.g., as a doll, puppet, or miniature toy car).

[0094] Figure 1A and Figure 1B An exemplary tap sequence is shown that allows a device user to indicate a selection from available responses displayed on the display 12a, 12b, 12c of the handheld device 10. For example, in an educational curriculum designed to help young children learn spelling and / or pronunciation, specifying such responses may be included in the machine-based interaction. In this example, the algorithm used to classify the tap locations may be informed that the possible results are limited to the three locations displayed by the device (i.e., a small number compared to typical applications, which may simplify the classification process).

[0095] Figure 1AThe handheld device 10 is shown being held by the right hand 14a of the device user (other than the hand, the rest of the user's body is not shown). The handheld device includes three spherical components 11a, 11b and 11c, which can be used as elements of camera-based tracking (not shown) of the device. Each of the three spherical components 11a, 11b and 11c includes a display 12a, 12b, 12c, which respectively project the letter "C" at 12a, "A" at 12b, and "T" at 12c; collectively forming the word "CAT". From Figure 1A and Figure 1B From this viewing angle, the grating covering the loudspeaker 13 in the handheld device can also be seen.

[0096] During a session using an electronic device, a young child may be asked to find the first letter C in the word "cat". Alternatively or in addition, the child may be asked (e.g., via speaker 13) to indicate the letter that can produce the phonetic "k" sound. Figure 1A As shown, the child uses the index finger 15 of the left hand 14b to strike or tap (i.e., via an up and down motion 16a) the top of the sphere 11a of the display 12a that contains the projected letter "C". A classification can then be performed on which sphere based on the acquired IMU data to determine whether the answer selected by the child is correct.

[0097] like Figure 1B As shown, the child may then be asked to find the next character "A" and / or the sound associated with the phonetic symbol "ae". The correct answer may be indicated by tapping anywhere on the middle sphere 11b of the handheld device 10. Figure 1B The child is shown striking the central sphere 11b using the same up and down motion 16b of the index finger 15 of the left hand 14b. Once struck, a classification process can be performed based on the acquired IMU data to determine which sphere was struck and, in turn, whether the answer indicated by the strike is correct.

[0098] Figure 2 The ability to use a single hand 24 to indicate a tap location on a handheld device 20 is shown. In this case, the handheld device 20 is held by the right hand of the device user. Figure 2 The device is hidden in the figure) and fingers 25b, 25c, 25d and 25e other than the thumb 25a are capable of mechanically stabilizing the entire device 20, allowing movement of the right thumb 25a (i.e., relative to the device 20) to include reaching the device speaker 23 and three spherical camera-based tracking components 21a, 21b and 21c, each of which includes a display 22a, 22b and 22c.

[0099] The area in the area of ​​the speaker 23 and the three spheres 21a, 21b and 21c is easily accessible by the thumb 25a. For most people (i.e., depending on the dexterity of the hand), it is also possible to tap the left side 27a of the device 20 and the area on most of the top surface 27b by curling the thumb 25a. A single finger 25b, 25c, 25d and 25e can be used to tap the side of the device 20. It is even possible to tap the back of the display 21a, 21b and 21c by manually rotating the device before tapping (i.e., still using a single hand), the right side of the phone (in the Figure 2 ) and / or the bottom of the handheld device 20 (not shown in the viewing perspective).

[0100] In response to an auditory prompt using device speaker 23 and / or a visual prompt projected by any of displays 22a, 22b, and 22c, the user may tap with thumb 25a to indicate a selected response. Figure 2 As shown, a simple up and down motion of the thumb produces a tap in the area of ​​the speaker 23. Alternatively or in addition, the device user can indicate a response and / or selection by tapping multiple times 26 in substantially the same area of ​​the device. Once tapped, a classification process can be performed based on the acquired IMU data to determine which area of ​​the device was tapped and / or the timing of the multiple taps.

[0101] Figure 3A and Figure 3B The handheld device itself 30 is shown being moved to strike or tap a selected area of ​​the device on an object or surface 37. The handheld device 30 is shown being held by a palm (obscured by the device) and fingers 35b, 35c, 35d and 35e (including the thumb 35a of the user's right hand 34). Each of the three spherical elements 31a, 31b and 31c attached to the device contains a display 32a, 32b and 32c. The leftmost display projects the word "dog" 32a. The center display 32b displays an image of an animal and the leftmost display 32c displays symbols representing different speech sounds. Figure 3A and Figure 3B The perforated cover on the embedded speaker within device 30 can also be seen in the viewing perspective view.

[0102] exist Figure 3AIn response to a visual prompt (e.g., on display 32b) or an auditory cue (e.g., via speaker 33), a device user may wish to indicate that the center display 32b displaying an animal is the selected response. The user may indicate the response by impacting 36a the bottom side of the center sphere 31b against a solid surface such as the edge of a table or desk 37. Once impacted, a classification process may be performed based at least in part on the acquired IMU data to determine which sphere was impacted (i.e., including identifying whether the bottom display surface was impacted). The classification process may also distinguish (e.g., based on the large peak amplitude and short duration of acceleration during the impact) whether the device 30 was impacted by a rigid object or a tap generated by a finger (i.e., a softer contact surface produces smaller movement and / or acceleration forces).

[0103] Then, if Figure 3B As shown, the desired response to the subsequent prompt may involve tapping the rightmost display 31c (i.e., displaying a symbol representing speech) against a solid surface 37. In this case, the handheld device 30, including its three displays 31a, 31b, 31c and speaker 33, is slightly rotated by the right hand 34 of the device user before performing an up and down tapping motion 36b against the corner of a table or desk 37. Once struck, based on the acquired IMU data, the classification process can determine that the outer edge of the rightmost display sphere 31c was struck against the solid surface 37 and that the device was moved (sideways) to produce the tap.

[0104] Figure 4A , Figure 4B and Figure 4C Examples of IMU accelerometer data collected when a handheld device is tapped are shown, showing some unique features in the data stream resulting from different forms of tapping and tapping in different locations. In each graph, data from a three-axis accelerometer is shown, with three orthogonal axes (denoted as X, Y, and Z) running roughly along the Figure 5 The axis orientations are shown in . The accelerometer measurements for each axis are sampled at approximately 6667 samples / second. Treating the accelerometer data as a three-dimensional vector, the magnitude of the acceleration |A| is also displayed and calculated according to the following formula

[0105]

[0106] Where X i , Y i and Z i represents the accelerometer samples in each of the three dimensions (ie, where "i" represents the sample index); X b , Y b and Z bThe so-called "baseline" values ​​for each of the same three dimensions are represented. The baseline values ​​can take into account factors such as electronic offsets of the sampling, and can be determined over periods of time when the accelerometer is not moving (e.g., by calculating an average to reduce the effects of noise). The three-dimensional acceleration directions can also be calculated from these data streams; however, these are not included in the Figure 4A , Figure 4B and Figure 4C Also not shown are the multi-dimensional IMU gyroscope data streams and the vectors of gravitational and / or magnetic forces pointing toward the Earth.

[0107] Figure 4A shows data acquired during tapping the left display of a handheld device using the index finger of the opposite hand (i.e., the hand opposite to the hand holding the device), similar to Figure 1A As described in Formula 1, the acceleration magnitude |A| on 41 is calculated using the acceleration trajectories in the X-axis on 40a, the Y-axis on 40b, and the Z-axis on 40c. Figure 4A 40a, the vertical scale of acceleration is not calibrated in absolute values; however, the values ​​in the X 40a, Y 40b and Z 40c dimensions can be compared relative to each other. The time bar at 43a represents 25 milliseconds.

[0108] The occurrence and time of a tap 42a is identified as a result of the acceleration magnitude 41 exceeding a predetermined threshold 42b. Such a threshold may take into account signal noise and the magnitude of acceleration resulting from taps versus non-tap device movement (e.g., performing other gestures). As described above, the accuracy of determining the timing of a tap may be improved by extrapolating samples in the region where the signal exceeds threshold 42a back to a baseline level. Among other unique features in the accelerometer data (e.g., Figure 4B and Figure 4C ), the initial acceleration of the tap on the Y axis is shown as a sharp rise in the positive direction 42d. As discussed further below, there is also a track feature 42c on the Z axis about 5 milliseconds after the initial tap contact, which may be the result of a break in contact between the finger and the left display of the handheld device.

[0109] Figure 4B Data acquired during tapping the right display of the handheld device using the index finger of the opposite hand is shown. According to Formula 1, the acceleration magnitude |A| at 45 is calculated using the acceleration trajectories on the X-axis at 44a, the Y-axis at 44b, and the Z-axis at 44c. Figure 4A and Figure 4B In FIG. 4 , the vertical scale of all acceleration traces 40a, 40b, 40c, 44a, 44b and 44c is the same. Figure 4BIn FIG. 4 , the time bar 43b represents 25 milliseconds. When the acceleration magnitude 45 exceeds a predetermined threshold value 46b, the occurrence and occurrence time 46a of a tap are identified.

[0110] exist Figure 4B In (with Figure 4A Conversely), the initial acceleration on the Y axis 44b after the tap at 46a is in the negative Y direction 46d. Figure 4A The initial positive acceleration 42d generated by the left display shown in FIG. Figure 4B The initial negative acceleration 46d behind the right display in FIG. 4 may be the result of the left and right displays being located on opposite sides of the Y axis (see Figure 5 ).

[0111] and Figure 4A similar, Figure 4B Also shown is a track feature 46c on the Z axis about 5 milliseconds after the initial tap contact. Some tracks collected during other taps (not shown) exhibit track features on different axes at about the same time after a tap with an amplitude close to that of the initial tap. Such track features may be the result of a break in contact between the finger and the device (e.g., different tapping forces generated during the separation process may represent mechanical changes).

[0112] Figure 4C Shown in Figure 3B The data is obtained during the process of tapping the rightmost display of the handheld device against a solid surface (i.e., a desktop) similar to the action represented in FIG. According to Formula 1, the acceleration trajectories of the X-axis 47a, the Y-axis 47b, and the Z-axis 47c are used to calculate the acceleration amplitude |A| at 48. Figure 4A and Figure 4B In comparison, the vertical scale of all acceleration data is approximately 60% larger (i.e., Figure 4C The peak amplitude is larger in Figure 4C 4, the time bar 43c represents a duration of 25 milliseconds. The occurrence and time of occurrence 49a of a tap is identified as a result of the acceleration magnitude 48 exceeding a predetermined threshold 49b.

[0113] In cases involving movement of a handheld device toward a solid surface, the acceleration of the device itself may be viewed in the data stream (e.g., at 49d). The peak amplitude resulting from such device movement may generally be smaller than the amplitude resulting from any form of tapping. Additionally, the peak amplitude of a tap against a rigid surface (e.g., a tabletop) (e.g., at 49e) may generally be larger than a tap generated using a finger (or other softer surface). Taps of a handheld device against a solid surface may also generate high frequency accelerations (e.g., reverberations) in the period following the tap (e.g., approximately 6 milliseconds at 49c). As the handheld device moves backward after impacting a solid surface, the amplitude |A| of the acceleration trajectory at 48 does not quickly return to a baseline level (e.g., at 49f) until the retraction movement of the device user is fully completed. This distinction (e.g., Figure 4C and Figure 4A and Figure 4B ) are examples of features that can be used to distinguish between tapping a handheld device with a finger or other object and tapping another object with the device.

[0114] Figure 5 Considerations for assigning a coordinate system for classifying the taps when determining the tap locations, in particular during numerical methods, are shown. Figure 5 The coordinate system shown consists of axes conventionally labeled X at 55a, Y at 55b, and Z at 55c, with an origin 54 located at the geometric center of the handheld device 50. In addition, rotational movement about the X axis at 56a, the Y axis at 56b, and the Z axis at 56c (i.e., corresponding to pitch, roll, and yaw) can be represented. Other coordinate systems are also possible, including, for example, using polar coordinates and / or placing the origin at the geometric or operational center of an IMU component located inside the device 50 (not shown). If one or more of the IMU components has the ability to sense the gravitational and / or magnetic attraction of the earth, the coordinate system of the handheld device 50 can additionally be represented relative to a vector 57b pointing in the direction of the earth's gravity 57a.

[0115] Tapping different locations on the handheld device 50 produces different translational and rotational movements (i.e., measured using a multi-dimensional accelerometer and gyroscope, respectively), which can be expressed using one or more such coordinate systems. Figure 4A , Figure 4B and Figure 4C As shown, the IMU data stream can represent the rotational and / or translational movements (i.e., "forward" mathematical methods) caused by taps at different (i.e., known) locations on the device 50. The difference between these movements (i.e., "reverse" mathematical methods) can then be used in an algorithmic strategy (e.g., a numerical, neural network-based algorithmic strategy) to classify the tap location.

[0116] For example, a knock at 52a on the left sphere 51a can obviously cause a rotational movement around the Y axis 55b in the negative direction 56b, combined with a rotational movement around the X axis 55a in the positive direction 56a. On the other hand, a knock at 52b on the center sphere 51b may produce a small rotational movement 56b around the Y axis 55b, but a larger torque is produced in the positive direction 56a around the X axis 55a. Further contrast, a knock at 52c on the right sphere 51c can obviously cause a rotational movement around the Y axis 55b in the positive direction 56b, combined with a rotational movement around the X axis 55a in the positive direction 56a. In still further contrast, a knock in the area of ​​the speaker at 53 of the handheld device 50 may only produce a small rotational movement around any axis, but can produce a measurable translational movement in the negative Z axis 55c direction. For another example, knocking the lower left side of the device at 58 may mainly and obviously cause a significant rotational movement around the Z axis 55c in the positive direction 56c.

[0117] Figure 6 60a, a user may grasp the handheld device 61a to enable tapping and view displays 62a, 62b, and 62c. Next 60b, an initial lighting pattern (e.g., comprising one dot 62a, two dots 62b, or three dots 62c) may be projected by display 61b and may be viewed by the device user. Then, at 60c, IMU data is acquired and analyzed (e.g., including the calculated movement amplitude) to determine whether the movement exceeds a threshold 61c (i.e., indicating the occurrence of a tap). If the threshold does not exceed 60d, further IMU data 61d is collected. If the IMU data exceeds a threshold or other criteria for determining that a tap 60d has occurred, data near the time of the tap determination is collected to perform a tap location classification 60e.

[0118] Figure 6 The use of a neural network 61e to perform a classification process 60e is shown. In this exemplary method, IMU data in the area near the time when the threshold is exceeded is input to the neural network. The most likely knock location (e.g., the leftmost spherical display 63) can be generated as a neural network output, and optionally, other attributes such as the degree of confidence in the most likely knock location and the measured force applied during the knock (e.g., related to the knock type and / or other knock attributes) can also be generated.

[0119] In addition, time data about the taps is registered 60f, including the time of the taps 61f and the interval since the last tap 64, as well as any activity patterns that are determined to be related to two or more previous taps. Based on the tap location and other tap attributes such as timing and tap force, a new display pattern 60g can be generated for the device display 61g. The entire process can then be repeated 60b using the new display pattern.

[0120] Figure 7 is a flow chart outlining exemplary steps for classifying a tap location on a handheld device in response to an auditory cue or sound presented using a device speaker. For example, the sound may include one or more words, one or more pronunciations, one or more speech sounds, one or more musical sounds, one or more animal sounds, one or more natural sounds, one or more ringtones, one or more beeps, and / or one or more alarms.

[0121] Initially 70a, the user may grasp the handheld device 71a so as to be able to tap and listen to the auditory prompt from the speaker 72. Next 70b, a sound pattern (in this case, a series of musical notes 71b) is broadcast on the speaker. The IMU data is then acquired 70c and analyzed 71c to determine whether the movement exceeds a threshold (i.e., indicating the occurrence of a tap). If the threshold is not exceeded 70d, the process of collecting IMU data is repeated 71d. If the IMU data exceeds a threshold or meets other criteria for determining that a tap 70d occurred, the data near the time of the tap determination is used to perform a tap location classification 70e.

[0122] Figure 7 The use of a numerical method 71e to perform the classification process 70e is shown. In this case, for example, a tap on the lower left side of the phone 73 by the index finger of the hand opposite to the hand used to grasp the device can be determined based on the dominant rotation signal in the XZ plane 71e (i.e., counterclockwise when viewed facing the upper part of the device). The most likely tap location is optionally calculated along with other attributes, such as confidence about the most likely tap location and the force applied during and after the tap (e.g., tap attributes).

[0123] Additional time data about the tap may be registered 70f, including the time of the tap 71f and the interval 74 since the previous tap, as well as determining any activity patterns when combined with the previous tap. Based on the newly determined tap location and other tap attributes, a new sound pattern 71g may be generated for broadcast 70g. The entire process may then be repeated 70b using the new auditory stimulus.

[0124] The disclosure of the above examples is presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Based on the above disclosure, many variations and modifications of the examples described herein will be apparent to those of ordinary skill in the art. It should be understood that various components and features described with specific examples may be added, deleted, and / or replaced with other examples, depending on the intended use of the examples.

[0125] In addition, when describing representative examples, the specification may have presented the method and / or process as a specific sequence of steps. However, in the case where the method or process does not rely on the specific sequence of steps described herein, the method or process should not be limited to the described specific sequence of steps. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims.

[0126] Although the invention is susceptible to various modifications and alternative forms, specific examples thereof have been shown in the drawings and described in detail herein. It should be understood that the invention is not limited to the specific forms or methods disclosed, but rather, the invention will cover all modifications, equivalents and alternatives falling within the scope of the appended claims.

Claims

1. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and at least one device display attached to the device body operably coupled to the device processor, wherein the device processor is configured to: generating, by the at least one device display, a first lighting pattern; acquiring inertial measurement data from the at least one inertial measurement unit; determining an initial tap location of the device user on one of the device body and the at least one device display based at least in part on the inertial measurement data; and Based at least in part on the initial tap location, a second illumination pattern is generated by the at least one device display. 2 . The device of claim 1 , wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes. The apparatus of claim 1 , wherein direction data and magnitude data are calculated from the inertial measurement data.

4. The device of claim 1, wherein the device processor is further configured to determine an orientation of the handheld device relative to one or both of the Earth's gravitational pull and the Earth's magnetic pull based on the inertial measurement data.

5. A device according to claim 1, wherein the initial tap at the initial tap location is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping the at least one device display with a finger of the second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, tapping a solid object with the handheld device, tapping the solid object with the at least one device display, tapping an additional handheld device with the handheld device, and tapping an additional handheld device display with the at least one device display.

6. The device of claim 1, further comprising a haptic unit that alerts a user of the device that the first lighting pattern has been generated on the at least one device display.

7. The device of claim 1, wherein the first illumination pattern comprises one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

8. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and a speaker within the device body operatively coupled to the device processor, wherein the device processor is configured to: generating a first sound through the speaker; acquiring inertial measurement data from the at least one inertial measurement unit; determining an initial tap location of the device user on the device body based at least in part on the inertial measurement data; and Based at least in part on the initial strike location, a second sound is generated by the speaker.

9. The device of claim 8, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

10. The device of claim 9, wherein the initial sound comprises one or more of one or more words, one or more pronunciations, one or more speech sounds, one or more musical sounds, one or more animal sounds, one or more natural sounds, one or more ringtones, one or more beeps, and one or more alarms.

11. The device of claim 9, wherein the initial tap at the initial tap location is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, and tapping a solid object with the handheld device.

12. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; At least one inertial measurement unit within the device body is operably coupled to the device processor, wherein the device processor is configured to: acquiring inertial measurement data from the at least one inertial measurement unit; Calculating direction data and amplitude data based on the inertial measurement data; determining an initial tap location of the device user on the device body based at least in part on one or both of the direction data and the magnitude data; and An action is performed based at least in part on the initial tap location on one or both of the device processor and a remotely connected processor.

13. The device of claim 12, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

14. The device of claim 12, wherein the initial tap at the initial tap location is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, and tapping a solid object with the handheld device.

15. The device of claim 12, wherein additional tap locations on the device body are determined based on additional inertial measurement data acquired from the at least one inertial measurement unit.

16. The device of claim 12, wherein the device further comprises one or both of a Wi-Fi communication module and a Bluetooth communication module configured to communicate between the device processor and the remotely connected processor.

17. The device of claim 16, wherein the action is further based on a location at which the device user was pointing on a touch-sensitive screen operably coupled to the remote processor when the initial tap was generated at the initial tap location.

18. The device of claim 12, wherein the device processor is further configured to determine an orientation of the handheld device relative to one or both of the Earth's gravitational pull and the Earth's magnetic pull based on the inertial measurement data.

19. The device of claim 18, wherein the action is further based on one or more of determining an initial occurrence time of the initial tap location, the one or more additional tap locations and determining additional occurrence times of the one or more additional tap locations, and an orientation of the handheld device.

20. The device of claim 12, wherein the system further comprises one or more buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor; each operably coupled to the device processor.

21. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and at least one device display attached to the device body operably coupled to the device processor, wherein the device processor is configured to: generating, by the at least one device display, a first lighting pattern; acquiring inertial measurement data from the at least one inertial measurement unit when a user taps the device in response to the first illumination pattern; determining an initial tap location of the device user on one of the device body and the at least one device display based solely on the inertial measurement data; and Based at least in part on the initial tap location, a second illumination pattern is generated by the at least one device display.

22. The device of claim 21, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

23. The apparatus of claim 21, wherein orientation data and magnitude data are calculated from the inertial measurement data.

24. The device of claim 21, wherein the device processor is further configured to determine an orientation of the handheld device relative to one or both of the Earth's gravitational pull and the Earth's magnetic pull based on the inertial measurement data.

25. A device according to claim 21, wherein the initial tap at the initial tap location is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping at least one device display with a finger of the second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, tapping a solid object with the handheld device, tapping the solid object with at least one device display, tapping an additional handheld device with the handheld device, and tapping an additional handheld device display with the at least one device display.

26. The device of claim 21, further comprising a haptic unit that alerts the device user that the first lighting pattern has been generated on the at least one device display.

27. The apparatus of claim 21, wherein the first illumination pattern comprises one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

28. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor; and a speaker within the device body operatively coupled to the device processor, wherein the device processor is configured to: generating a first sound through the speaker; acquiring inertial measurement data from the at least one inertial measurement unit when the user taps the device in response to the first sound; determining an initial tap location of the device user on the device body based solely on the inertial measurement data; and Based at least in part on the initial strike location, a second sound is generated by the speaker.

29. The device of claim 28, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

30. The device of claim 29, wherein the initial sound comprises one or more of one or more words, one or more pronunciations, one or more speech sounds, one or more musical sounds, one or more animal sounds, one or more natural sounds, one or more ringtones, one or more beeps, and one or more alarms.

31. A device according to claim 29, wherein the initial tapping at the initial tapping location is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, and tapping a solid object with the handheld device.

32. A handheld device for device-user interaction, comprising: a device body configured to be held by a first hand of a user of the device; electronic circuitry within the device body, which includes a device processor; at least one inertial measurement unit within the device body operably coupled to the device processor, wherein the device processor is configured to: acquiring inertial measurement data from the at least one inertial measurement unit; Calculating direction data and amplitude data based on the inertial measurement data; determining, based on one or both of the direction data and the magnitude data, an initial tap location of the device user on the device body when the user generates an initial tap using the device; and An action is performed based at least in part on the initial tap location on one or both of the device processor and a remotely connected processor.

33. The device of claim 32, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

34. A device according to claim 32, wherein the initial tap is generated by the device user in one of the following ways: tapping the handheld device with a second hand, tapping the handheld device with a finger of a second hand, tapping the handheld device with a finger of a first hand, tapping the handheld device with a body part, and tapping a solid object with the handheld device.

35. The device of claim 32, wherein additional tap locations on the device body are determined based on additional inertial measurement data obtained from the at least one inertial measurement unit.

36. The device of claim 32, wherein the device further comprises one or both of a Wi-Fi communication module and a Bluetooth communication module configured to communicate between the device processor and the remotely connected processor.

37. The device of claim 36, wherein the action is additionally based on a location at which the device user was pointing on a touch-sensitive screen operably coupled to the remote processor when the initial tap was generated at the initial tap location.

38. The device of claim 32, wherein the device processor is further configured to determine an orientation of the handheld device relative to one or both of the Earth's gravitational pull and the Earth's magnetic pull based on the inertial measurement data.

39. A device according to claim 38, wherein the action is additionally based on one or more of determining an initial occurrence time of the initial tap location, determining one or more additional tap locations based on one or more additional taps, and determining additional occurrence times of the one or more additional tap locations and an orientation of the handheld device.

40. The device of claim 32, wherein the system further comprises one or more buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor; each operably coupled to the device processor.

41. A method for human interaction using a handheld device, the handheld device comprising a device processor, at least one device display attached to a device body and operably coupled to the device processor, and at least one inertial measurement unit within the device body operably coupled to the device processor, the method comprising: generating an illumination pattern on the at least one device display; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, direction data and magnitude data based on the inertial measurement data; determining, by the device processor, a tap location on the device body when the human generates a tap with the handheld device based on one or both of the direction data and the magnitude data; and An action is performed by one or both of the device processor and a remotely connected processor based at least in part on the tap location.

42. The method of claim 41, wherein the illumination pattern comprises one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.

43. A method according to claim 41, wherein the tapping is generated by the human in one of the following ways: tapping the handheld device held by the first hand with a second hand, tapping the handheld device with a finger of the second hand, tapping the at least one device display with a finger of the second hand, tapping the handheld device with a finger of the first hand, tapping the handheld device with a body part, tapping a solid object with the handheld device, tapping the solid object with the at least one device display, tapping an additional handheld device with the handheld device, and tapping an additional handheld device display with the at least one device display.

44. A method according to claim 41, wherein the device processor is also configured to determine the orientation of the handheld device relative to one or both of the earth's gravitational attraction and the earth's magnetic attraction based on the inertial measurement data, and wherein the action is also based on the orientation of the handheld device.

45. The method of claim 41, wherein the handheld device further comprises a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the lighting pattern is being generated on the at least one device display.

46. ​​The method of claim 41, wherein the handheld device further comprises a device speaker operably coupled to the device processor, the method further comprising broadcasting one or more sounds through the device speaker to alert the human being that the lighting pattern is being generated on the at least one device display.

47. A method for human interaction using a handheld device, the handheld device comprising a device processor, a device speaker within a device body operably coupled to the device processor, and at least one inertial measurement unit within the device body operably coupled to the device processor, the method comprising: generating one or more sounds on the device speaker; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, direction data and magnitude data based on the inertial measurement data; determining, by the device processor, a tap location on the device body when the human generates a tap with the handheld device based on one or both of the direction data and the magnitude data; and An action is performed by one or both of the device processor and a remotely connected processor based at least in part on the tap location.

48. The method of claim 47, wherein the one or more sounds include one or more of one or more words, one or more pronunciations, one or more voices, one or more musical sounds, one or more animal sounds, one or more natural sounds, one or more ringtones, one or more beeps, and one or more alarms.

49. A method according to claim 47, wherein the tapping is generated by the human in one of the following ways: tapping the handheld device held by the first hand with a second hand, tapping the handheld device with a finger of the second hand, tapping at least one device display with a finger of the second hand, tapping the handheld device with a finger of the first hand, tapping the handheld device with a body part, tapping a solid object with the handheld device, tapping the solid object with at least one device display attached to the device body, tapping an additional handheld device with the handheld device, and tapping an additional handheld device display with the at least one display.

50. The method of claim 47, wherein the handheld device further comprises a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the one or more sounds are being generated on the device speaker.

51. A method for human interaction using a handheld device, the handheld device comprising a device processor and at least one inertial measurement unit within a device body operably coupled to the device processor, the method comprising: acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit; calculating, by the device processor, direction data and magnitude data based on the inertial measurement data; determining, by the device processor, an initial tap location on the device body when the human generates an initial tap with the handheld device based on one or both of the direction data and the magnitude data; and An action is performed by one or both of the device processor and a remotely connected processor based at least in part on the initial tap location.

52. The method of claim 51, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers, and one or more gyroscopes.

53. A method according to claim 51, wherein the initial tap is generated by the human in one of the following ways: tapping a handheld device held by a first hand with a second hand, tapping the handheld device with a finger of a second hand, tapping at least one device display with a finger of the second hand, tapping the handheld device with a finger of the first hand, tapping the handheld device with a body part, tapping a solid object with the handheld device, tapping the solid object with at least one device display attached to the device body, tapping an additional handheld device with the handheld device, and tapping an additional handheld device display with the at least one display.

54. The method according to claim 51 also includes determining, by the device processor, one or more additional taps at one or more additional tap locations on the device body based on additional inertial measurement data obtained from the at least one inertial measurement unit, and wherein the action is also based on the one or more additional tap locations.

55. The method of claim 54, further comprising determining, by the device processor, one or more tap intervals between the initial tap and the one or more additional taps, and wherein the action is also based on an initial occurrence time of the initial tap and one or more of the at least one tap intervals.

56. The method of claim 51 further comprises determining, by the device processor based on the inertial measurement data, an orientation of the handheld device relative to one or both of the Earth's gravitational pull and the Earth's magnetic pull, and wherein the action is also based on the orientation of the handheld device.

57. The method of claim 51 , wherein the handheld device further comprises a device haptic unit operably coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the inertial measurement data is about to be acquired.

58. A method according to claim 51, wherein the handheld device further includes one or both of a Wi-Fi communication module operably coupled to the device processor and a Bluetooth communication module operably coupled to the device processor, and the communication modules communicate between the device processor and the remotely connected processor.

59. The method of claim 58, wherein: The action is also based on a screen location pointed to by the human on a touch-sensitive screen operably coupled to the remote processor when the initial tap was generated.

60. The method of claim 51, wherein the handheld device additionally includes one or more buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical cardiac sensor, and an electrical cardiac sensor; each operably coupled to the device processor.

Images