Game having hand motion control

Abstract

To provide a game having hand motion control.SOLUTION: Motion of a wrist band is used to control a game. There is provided a method including the processes of: receiving a first wireless signal from a first device; receiving a second wireless signal from a second device; determining a first player identifier from the first wireless signal; determining a second player identifier from the second wireless signal; displaying a message for requesting identification of himself / herself with a player; receiving an instruction of an identifier of a third player via a tactile input section; determining that the identifier of the third player matches the identifier of the first player; receiving a third wireless signal from the first device; interpreting the third wireless signal as a command in a gamble game; and executing a command in the gamble game.SELECTED DRAWING: Figure 14b

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. patent application Ser. No. 11 / 754,944, filed May 29, 2007, entitled "Game With Hand Motion Control," the entirety of which is incorporated herein by reference. This application relates to games with hand motion control. [Background technology]

[0002] This application relates to games with hand motion controls. Summary of the Invention [Means for solving the problem]

[0003] The present invention provides a system for receiving a first wireless signal from a first device, receiving a second wireless signal from a second device, determining a first player identifier from the first wireless signal, determining a second player identifier from the second wireless signal, displaying a message requesting a player to identify themselves, receiving an indication of a third player identifier via a tactile input, and determining that the third player identifier matches the first player identifier; A method is provided, comprising the steps of receiving a third wireless signal from the first device, interpreting the third wireless signal as a command in a gambling game, and executing the command in the gambling game. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 illustrates a gaming system according to some embodiments. [Figure 2] FIG. 2 illustrates a communication network according to some embodiments. [Figure 3]FIG. 3 illustrates a gaming service provider in communication with a gaming communication device according to some embodiments. [Figure 4] FIG. 4 illustrates a gaming network according to some embodiments. [Figure 5] FIG. 5 illustrates a gaming system according to some embodiments. [Figure 6] FIG. 6 illustrates a wireless gaming system according to some embodiments. [Figure 7] FIG. 7 illustrates a mobile gaming device with promotional content according to some embodiments. [Figure 8] FIG. 8 is a block diagram of a gaming system according to some embodiments. [Figure 9] FIG. 9 is a block diagram of a payment system forming part of the gaming system shown in FIG. 8, according to some embodiments. [Figure 10] FIG. 10 is a schematic diagram of a mobile gaming device of the gaming system shown in FIG. 8, according to some embodiments. [Figure 11a] FIG. 11(a) is a flow diagram of a method for a player to use a mobile gaming device, according to some embodiments. [Figure 11b] FIG. 11(b) is a flow diagram of a particular method of use of a mobile gaming device by a player, according to some embodiments. [Figure 12] FIG. 12 is a flow diagram of a method for use of a mobile gaming device by a gaming service operator, according to some embodiments. [Figure 13] FIG. 13 is a flow diagram of a method of using a mobile gaming device, according to some embodiments. [Figure 14a] 1 illustrates several camera-based embodiments. [Figure 14b] 1 illustrates several 3-D (three-dimensional) detection embodiments. [Figure 14c] Some embodiments are shown with a two camera "binocular" stereo camera. [Figure 14d] 1 illustrates some steps according to some embodiments. [Figure 14e]1 illustrates a process for color mapping according to some embodiments. [Figure 15] 1 illustrates the hardware components of an implementation of a multi-camera control system and its physical layout according to some embodiments. [Figure 16A] 16 illustrates the geometric relationship between the camera and the various image regions of FIG. 15 according to some embodiments. [Figure 16B] FIG. 15 shows an image captured by one of the cameras, according to some embodiments. [Figure 17] FIG. 1 is a flow diagram illustrating a process executed within a microcomputer program associated with a multi-camera control system, according to some embodiments. [Figure 18] FIG. 18 is a flow diagram illustrating in more detail a portion of the process shown in FIG. 17, particularly the process related to detecting and extracting the location of an object from image signals captured by a camera, according to some embodiments. [Figure 19A] 19 shows sample image data presented as a grayscale bitmap image captured by a camera and generated by part of the process shown in FIG. 18, according to some embodiments. [Figure 19B] 19 shows sample image data presented as a grayscale bitmap image generated by part of the process shown in FIG. 18, according to some embodiments. [Figure 19C] 19 shows sample image data presented as a grayscale bitmap image generated by part of the process shown in FIG. 18, according to some embodiments. [Figure 19D] 19 shows sample image data presented as a grayscale bitmap image generated by part of the process shown in FIG. 18, according to some embodiments. [Figure 19E]19 shows sample data presented as a binary bitmap image identifying pixels in the sample that are likely to belong to a tracked object, generated by part of the process shown in FIG. 18, according to some embodiments. [Figure 20] FIG. 19B is a flow diagram illustrating in more detail a portion of the process described in FIG. 18 , in accordance with some embodiments, particularly the process involved in classifying and identifying an object being tracked given a map of pixels identified as likely to belong to that object, e.g., given the data shown in FIG. 19E. [Figure 21A] FIG. 19E shows sample data presented as a binary bitmap image, along with an identification of the data sample that the process shown in FIG. 20 selected as belonging to the object in this sample, according to some embodiments. [Figure 21B] The process outlined in Figure 20 shows sample data presented in Figure 19E presented as a bar graph, along with an identification of data samples selected as belonging to an object, according to some embodiments. Specific points in the graph are identified. [Figure 21C] 20 shows different sets of sample data presented as binary bitmap images, along with identification of the object and the data samples selected by the process shown in FIG. 20 as belonging to key parts of that object in the sample, according to some embodiments. [Figure 22] 19 is a flow diagram illustrating in more detail a portion of the process shown in FIG. 18, in particular the process involved in generating and maintaining a description of background regions occluded by objects, according to some embodiments. [Figure 23A] According to some embodiments, Equation 3 shows the geometry on which it is based, i.e., the angle that defines the position of an object in the camera's field of view given the position on the image plane where the object is detected. [Figure 23B] Equations 4, 5, and 6 illustrate the underlying geometry, i.e., the relationship between the camera position and the object being tracked, according to some embodiments. [Figure 24] 8 is a graph illustrating the amount of attenuation that may be applied to coordinates given a change in object position to refine Equation 8, i.e., position, according to some embodiments. [Figure 25A] 1 is an example of an application program in which an object of interest is controlled by a system that controls a screen pointer in two dimensions, according to some embodiments. [Figure 25B] 25B illustrates a mapping between real-world coordinates and screen coordinates used by the application program of FIG. 25A according to some embodiments. [Figure 26A] 1 is an example of an application program controlled by a multi-camera control system in which an object of interest controls a screen pointer within a three-dimensional virtual reality environment, according to some embodiments. [Figure 26B] 1 is an example of an application program controlled by a multi-camera control system in which an object of interest controls a screen pointer within a three-dimensional virtual reality environment, according to some embodiments. [Figure 27A] 1 illustrates a division of an area into detection surfaces used by a gesture detection method for identifying gestures that may be associated with an intent to activate, according to some embodiments. [Figure 27B] 1 illustrates a division of a region of interest into detection boxes used by a gesture detection method to identify gestures that may be associated with selecting a cursor direction, according to some embodiments. [Figure 27C] 10 illustrates an alternative division of a region into direction detection boxes used by gesture detection to identify gestures that may be associated with selecting a cursor direction, according to some embodiments. [Figure 27D] 27D illustrates the relationship of adjacent sections of FIG. 27C in more detail, according to some embodiments. [Figure 28] 1 illustrates an appearance of a device according to some embodiments with the device in a neutral position. [Figure 29] 29 illustrates an example of the internal structure of the implementation of FIG. 28, according to some embodiments. [Figure 30] 1 is a flow diagram illustrating a method according to another example implementation, in accordance with some embodiments. [Figure 31A] FIG. 31A shows an example of tilt regions defined about a neutral axis, according to some embodiments. [Figure 31B] FIG. 31B shows an example of a tilt region defined about a neutral axis, according to some embodiments. [Figure 31C] FIG. 31C shows an example of a tilt region defined about a neutral axis, according to some embodiments. [Figure 31D] FIG. 31D shows an example of a tilt region defined about a neutral axis, according to some embodiments. [Figure 32] 1 illustrates an exterior top view of an exemplary device according to another exemplary implementation, according to some embodiments. [Figure 33A] FIG. 33A shows an exemplary indicator, according to some embodiments. [Figure 33B] FIG. 33B shows an exemplary indicator, according to some embodiments. [Figure 33C] FIG. 33C shows an exemplary indicator, according to some embodiments. [Figure 33D] FIG. 33D shows an exemplary indicator, according to some embodiments. [Figure 33E] FIG. 33E shows an exemplary indicator, according to some embodiments. [Figure 34A] 34A and 34B show front and side views, respectively, of the device of FIG. 32 shown in a neutral position, according to some embodiments. [Figure 34B] 34B shows front and side views, respectively, of the device of FIG. 32 shown in a neutral position, according to some embodiments. [Figure 35A]35A-35C show front views of the apparatus of FIG. 32 when the apparatus is operated in a negative roll orientation and a positive roll orientation, respectively, according to some embodiments. [Figure 35B] 35B shows front views of the apparatus of FIG. 32 when the apparatus is operated in a negative roll orientation and a positive roll orientation, respectively, according to some embodiments. [Figure 36A] 36A-36C show side views of the device of FIG. 32 when the device is operated in a positive pitch orientation and a negative pitch orientation, respectively, according to some embodiments. [Figure 36B] 36B shows side views of the device of FIG. 32 when the device is operated in a positive pitch orientation and a negative pitch orientation, respectively, according to some embodiments. [Figure 37] 10 is a table illustrating one possible mapping of device orientations used to output signals corresponding to the letter and case that is output when a control is selected, according to some embodiments. [Figure 38A] FIG. 38A shows a menu of symbols displayed according to another exemplary implementation, according to some embodiments. [Figure 38B] FIG. 38B shows a menu of symbols displayed according to another exemplary implementation, according to some embodiments. [Figure 39] FIG. 1 is an external view of a game system F1 according to some embodiments. [Figure 40] FIG. 40 is a functional block diagram of the game device F3 shown in FIG. 39. [Figure 41] FIG. 40 is a perspective view showing the appearance of the controller F7 shown in FIG. 39. [Figure 42] 42 is a perspective view showing a state of the connection cable F79 of the controller F7 shown in FIG. 41 when it is about to be connected to or disconnected from the core unit F70. FIG. [Figure 43] FIG. 42 is a perspective view of the core unit F70 shown in FIG. 41 as seen from above and behind. [Figure 44] 42 is a perspective view of the core unit F70 shown in FIG. 41 as viewed from the front side of the bottom surface. FIG. [Figure 45] FIG. 42 is a perspective view showing a state in which the upper casing of the core unit F70 shown in FIG. 41 is removed. [Figure 46] 42 is a perspective view showing a state in which the lower casing of the core unit F70 shown in FIG. 41 is removed. FIG. [Figure 47] FIG. 42 is a perspective view showing a first example of the subunit F76 shown in FIG. 41. [Figure 48] FIG. 48 is a perspective view of the subunit F76 shown in FIG. 47 with the upper casing removed. [Figure 49A] FIG. 49A is a top view of a second example of subunit F76 shown in FIG. [Figure 49B] FIG. 49B is a bottom view of the second example of subunit F76 shown in FIG. [Figure 49C] FIG. 49C is a left side view of the second example of subunit F76 shown in FIG. [Figure 50] FIG. 42 is a perspective view of the subunit F76 shown in FIG. 41 as seen from the upper front side. [Figure 51] FIG. 42 is a top view showing an example of a first modification of the subunit F76 shown in FIG. 41. [Figure 52] FIG. 42 is a top view showing an example of a second modification of the subunit F76 shown in FIG. 41. [Figure 53] FIG. 42 is a top view showing an example of a third modification of the subunit F76 shown in FIG. 41. [Figure 54] FIG. 42 is a top view showing an example of a fourth modification of the subunit F76 shown in FIG. 41. [Figure 55] FIG. 42 is a block diagram showing the structure of a controller F7 shown in FIG. 41. [Figure 56] FIG. 42 is a diagram showing the state of the game being controlled using the controller F7 shown in FIG. 41. [Figure 57]10 shows an exemplary state of a player holding the core unit F70 in his right hand, as viewed from the front side of the core unit F70. [Figure 58] 10 shows an exemplary state of a player holding the core unit F70 in his right hand, as viewed from the left side of the core unit F70. [Figure 59] 10 is a schematic diagram showing the viewing angle of the LED module F8L, the viewing angle of the LED module F8R, and the viewing angle of the image pickup element F743. [Figure 60] 10 shows an exemplary state of a player holding the subunit F76 in his left hand, as viewed from the right side of the subunit F76. [Figure 61] 10 shows an exemplary game image displayed on the monitor F2 when the game device F3 is playing a shooting game. DETAILED DESCRIPTION OF THE INVENTION

[0005] In various embodiments, a player may use motion as input to a game played on a mobile gaming device. The game may be a gambling game, such as a live data game, a slot machine game, a game of roulette, a game of craps, or any other gambling game. The player may place a bet on the game and may be in a position to win money depending on the outcome of the game. The player may be in a position to lose money on the game.

[0006] Motions used as input may include motions of the mobile gaming device itself. Thus, a player may tilt, shake, move, rotate, or otherwise move the mobile gaming device. Such movements of the mobile gaming device may be interpreted by hardware sensors and / or software as commands or instructions for playing a game. Thus, one motion may be seen as a start signal for a game or as a signal for a cashout.

[0007] In various embodiments, a player may be provided with audio feedback. This audio feedback may be provided according to a motion made by the player or according to a motion recognized by the mobile gaming device. The audio feedback may be provided during a motion being made by the player. This audio feedback may enhance the player's gaming experience by providing the player with sounds similar to those that would be heard while playing a game on a physical gaming table or a stand-alone gaming device such as a slot machine. The audio feedback may provide information to the player. The audio feedback may inform the player that a motion made by the player was recognized as a command or that a motion made by the player was not recognized as a command.

[0008] In various embodiments, the player may be provided with force feedback or haptic feedback. The mobile gaming device may create haptic sensations using springs, motors, resistors, or other devices that may create motion, pressure, heat, or other tactile or other sensations. The haptic feedback may give the player the sensation of shaking the mobile gaming device in the player's hand and, for example, rolling dice.

[0009] In various embodiments, a player may have a wristband. The wristband may include a motion sensor, such as an accelerometer, to detect motion. The player may move the hand wearing the wristband in a particular manner to issue commands to the game. In various embodiments, the wristband may provide tactile feedback.

[0010] (wristband / bracelet) In various embodiments, a player may wear a bracelet, watch, wristband, or other device around the player's wrist. The wristband may include one or more of the following: (a) a processor (e.g., a semiconductor processor); (b) a power source (e.g., a battery); (c) a motion sensor (e.g., an accelerometer; e.g., a gyroscope; e.g., a camera for measuring motion based on changing visual images); (d) a transmitter (e.g., an antenna); (e) a receiver (e.g., an antenna); (f) a memory (e.g., semiconductor memory); (g) a display device (e.g., a liquid crystal display screen); (h) a speaker (e.g., for transmitting audio output); (i) a tactile output device.

[0011] (The wristband records your movements.) In various embodiments, the wristband may track motions made by a player wearing the wristband. For example, motion sensors within the wristband may detect acceleration, change in position, change in orientation, angular displacement, path, trajectory, or any other component of motion. The wristband may track motions of the hand or wrist on which the wristband is worn. The wristband may store data representing the motions. Such data may be stored, for example, in memory within the wristband. The wristband may transmit an indication of the motions made to another device, for example, to a mobile gaming device, a stationary gaming device, or a casino server.

[0012] In various embodiments, the wristband may store or transmit raw data, such as data indicative of any readings received from a motion sensor. In various embodiments, the wristband may translate this raw data into more condensed or advanced data. For example, a series of readings from a motion sensor in the bracelet may be translated into a command. That is, a player wearing the wristband may make a motion to give a command. The wristband may then store that command rather than the exact position of the wristband as a function of time. The wristband may transmit the command to another device, for example, via a transmitter on the wristband.

[0013] (Motions constitute commands in games) In various embodiments, wristband motions may be interpreted as commands in a game. A player may, for example, move their hand up and down to start the spinning of reels in a slot machine game. A player may also move their hand in a manner that represents a command to (a) check out; (b) keep cards in video poker; (c) discard cards in video poker; (d) double down in blackjack; (e) select one of several options in a bonus round; (f) place a bet of a particular size; (g) view a list of game instructions; (h) start a bonus round; (i) select a payline to play, or perform any other command in a game. The wristband may store a table that associates particular motions with particular game commands. Upon receiving a sensor reading indicating a particular motion, the wristband may look up the motion corresponding to that command in the table. The wristband may then transmit the command to a mobile gaming device, a stationary gaming device, or another device, such as a casino server. The casino server may relay the command to another device, such as a stationary gaming device or a mobile gaming device, which, in various embodiments, may then be executed or followed in the game.

[0014] (The wristband communicates with the mobile gaming device.) In various embodiments, the wristband may communicate with a mobile gaming device. The wristband may have an antenna and receiver for this purpose. The mobile gaming device may also have an antenna and receiver for communicating with other devices. The mobile gaming device and wristband may communicate via various protocols, such as Bluetooth, Wi-Fi, or any other protocol.

[0015] (The wristband controls other devices) The wristband may be in communication with a mobile gaming device, a stationary gaming device, or any other device. The wristband may detect player motion, such as the motion of the player's hand. The wristband may interpret this motion as a command for the device with which the wristband is communicating. The wristband may send commands to that device, which in turn may cause the other device to follow the command. In some embodiments, the wristband captures raw data, such as a series of positions of the player's wrist as a function of time. This raw data is sent to another device. The other device then interprets this raw data as commands.

[0016] (Communicating with multiple devices simultaneously) In various embodiments, the wristband may be in communication with two or more devices. The wristband may be in communication with two or more devices simultaneously. The wristband may transmit a single signal that may be received by both a first device and a second device. For example, a command transmitted by the wristband may be received by a first slot machine and a second slot machine. In some embodiments, the first device and the second device may emit signals substantially simultaneously. The wristband may receive both signals.

[0017] In some embodiments, a player may identify himself or herself to two or more devices, such as two or more stationary gaming devices. The player may provide some proof of identity, such as a player tracking card, biometrics, or a device (e.g., a wristband), with an identifier (e.g., a unique identifier) ​​that can be tied to the player. The player may authorize or allow communication between the player's wristband and the two or more devices. As part of the authorization, the player may agree to play a game on each of the two or more devices. Thus, in some embodiments, the player may authorize the two or more devices to interpret signals coming from the player's wristband as command signals to be used in the game. In some embodiments, the player may present his or her wristband to two or more devices. For example, the player may bring the player's wristband within a few inches of an RFID reader on a slot machine. The slot machine may receive signals from the RFID tag on the wristband. The device may then recognize commands received from the presented wristband but may not recognize commands received from other wristbands. Thus, the device may accept commands from the wristband for some period of time. In various embodiments, commands may be accepted until some stop command is received, until no further commands are detected (e.g., if the wristband is turned off or taken out of range of the device), until a certain amount of time has passed, or until some other stop condition occurs. To resume providing motion-based commands to the device, the player may once again allow reception and use of commands from their wristband. For example, the player may present their wristband.

[0018] In various embodiments, a player may be involved in play at two or more gaming devices simultaneously. The player may initiate a motion, and an indication of such motion (e.g., a command derived from such motion) may be transmitted to two or more gaming devices. Each of the two or more gaming devices may execute the command. Thus, in some embodiments, a player can conveniently play two or more games simultaneously while avoiding the need to repeat commands for each individual game. For example, a player may use a single flick of the wrist to start a game at each of two slot machines.

[0019] In some embodiments, a first device may receive data (e.g., motion data) from the wristband. The first device may interpret the data as commands and play games based on these commands. A second device may receive the same data from the wristband. The second device may transmit the data (or its interpretation) to the player's friends or other parties, allowing them to follow what the player is doing. The second device may transmit indications of game outcomes, payouts, and other occurrences related to games played by the player to the player's friends or other parties. In some embodiments, a player may play several games simultaneously using motions from his or her wristband. Data from those games (e.g., outcomes) may be transmitted to a casino server or another device. The data may be available for viewing by other parties, such as the player's friends, or by others who will play their own games using random occurrences in the player's games (e.g., others may bet based on the outcomes that occur in the player's games).

[0020] In various embodiments, a player may play on two gaming devices simultaneously. However, each command generated by a player (e.g., through a motion) may apply to only one gaming device at a time. For example, a player may generate a first command that applies only to a first game on a first gaming device. The player may then generate a second command that applies only to a second game on a second gaming device. The player may then generate a third command that applies only to the first game on the first gaming device. In various embodiments, the two gaming devices may each be controllable by their own set of motion commands, with little or no overlap between the motions used for the commands. Thus, for example, a motion generated by a player may correspond to a command valid on one of the gaming devices but not on the other gaming device. A different motion may not correspond to a command valid on the first gaming device but may correspond to a command valid on the second gaming device.

[0021] (time when no data stream is received from the wristband) In various embodiments, a device may be within communication range of a wristband transmitting data, but the device may not be able to receive the data, or the device may not be able to interpret the data, or the device may not be able to use the data. The device may be a mobile gaming device or a stationary gaming device, such as a slot machine. A device may not be able to use data from a wristband if one or more of the following apply: (a) the player wearing the wristband has not identified themselves to the device; (b) the player wearing the wristband has not presented identification to the device; (c) the wristband is sending commands that the device does not understand; (d) the player wearing the wristband has not made at least some physical contact with the device (e.g., pressing a button on the device); (e) the player has not informed the device that it should expect motion commands from the wristband; (f) the device is currently accepting motion commands from a different wristband; (g) the player does not have a high enough credit balance to play a game on the device (e.g., the player has a credit balance of zero); (h) the player has not made physical contact with the device for a predetermined period of time (e.g., the player has not physically pressed a button on the gaming device in the last 10 minutes); or if any other situation applies.

[0022] (Biometrics as input for games) In various embodiments, the wristband may sense a player's pulse, temperature, skin conductivity, moisture level, electrical field (e.g., from nerve pulses), muscle tone, or any other biometric signal. This signal may be translated into a number. For example, a numerical temperature reading in degrees Fahrenheit may be used as a seed for a random number generator, which is then used to generate outcomes in the game.

[0023] In various embodiments, a biometric read received at a wristband may indicate that the wristband is still being worn. If the wristband detects a pulse, for example, the wristband or another device may infer that the wristband is being worn by a player and has not yet been removed. In various embodiments, a mobile gaming device, a stationary gaming device, or another device may act based on signals received from a wristband only if the wristband is currently being worn (or appears to be being worn based on biometric signals received from the wristband). In some embodiments, if there is an interruption in the biometric signals received at the wristband (e.g., the wristband no longer detects a pulse), the wristband may transmit a signal to a casino server or to some other device. This signal may indicate that there has been an interruption in the biometric signals detected at the wristband. Accordingly, the casino server may instruct other devices not to follow commands or signals received from the wristband until the wristband is re-established with the player. In some embodiments, the wristband must be re-established on the player in the presence or with the assistance of a casino representative before the signal from the wristband can be honored by another device. In some embodiments, if there is an interruption in the biometric signal detected by the wristband, the wristband may send a signal to page medical personnel. For example, the wristband may send a signal to the casino server indicating that a pulse is no longer being detected.

[0024] (The wristband broadcasts data that identifies the user.) In various embodiments, the wristband may transmit or broadcast data that identifies the player wearing the wristband. The wristband may broadcast a player tracking card number, the player's name, a player alias, the player's room number, the player's credit card number, or any other information about the player that may be used to identify the player. In some embodiments, the wristband may transmit a signal derived from a biometric reading. For example, the wristband may broadcast a signal derived from a pulse or electrocardiogram reading obtained from the player. This biometric reading may serve to uniquely identify the player.

[0025] In various embodiments, a signal broadcast from the wristband and identifying the player may grant certain privileges to the player wearing the wristband. The player's hotel room door may be unlocked remotely (e.g., the door may be unlocked from a key or other device without the need for physical contact). When the hotel room door receives a signal from the player's wristband identifying the player, the hotel room door may unlock. The player may be allowed to gamble at a particular gaming device. The player may be allowed to enter a particular area of ​​the casino based on the identity provided by the player's wristband. In various embodiments, the wristband may provide the player with an identifier to allow the player to receive access to a balance of funds or another financial account. The player may use the funds, for example, to gamble or make purchases. For example, the player may approach a gaming device. The player may have an account with a positive balance of funds stored with a casino server. When the player's wristband transmits a player identifier to a slot machine, the slot machine may receive the identifier and transmit an indication of the identifier to the casino server. The casino server may then allow the player to access the player's funds. Some or all of the player's funds may then be made available to the gaming device (e.g., in the form of a credit balance). The player may then use the funds to play games.

[0026] In various embodiments, a wristband may be power-constrained due to a small amount of available volume within the wristband that contains a battery or other power source. The wristband may take various steps to conserve power. In some embodiments, the wristband may periodically transmit a signal to another device, such as a mobile gaming device or a stationary gaming device. For example, the wristband may transmit a signal consisting of a series of bits to the mobile gaming device every 50 milliseconds. This signal may include data or information describing the motion produced by the wristband since the last signal transmission. In various embodiments, the time between signal transmissions may vary depending on what data or information needs to be transmitted by the wristband. For example, if the wristband is motionless for a period of time, the time between signal transmissions may be increased to 200 milliseconds. If the wristband begins to move again, the time between signal transmissions may be reduced again to 50 milliseconds. Thus, in various embodiments, the time between signals transmitted by the wristband may vary based on the motion of the wristband and / or based on motion detected by the wristband. In various embodiments, the amount of time between when a signal is transmitted by the wristband may vary based on the amount of information the wristband must communicate to another device. For example, if a player is actively involved in a game, the wristband may transmit signals frequently. If a player is not actively involved in a game (e.g., the player has not started playing a game on a stationary or mobile gaming device; e.g., the player is not in an area where gaming is permitted), the wristband may transmit signals less frequently. In various embodiments, if the wristband is not moving, the wristband may periodically transmit a short, simple signal indicating that the wristband is still active or still available for use. However, this signal may indicate that the wristband is not currently being used or is not being used for a game.

[0027] In various embodiments, the wristband may derive power or energy from the motion of the wearer's arm or from other motions of the wearer. The wristband may derive energy from its own motion, which may be caused by the motion of the arm to which the wristband is attached. Devices for harnessing electrical energy from motion may include piezoelectric devices or mechanical rotary magnetic generators. Power sources such as those used in Fossil kinetic watches or Ventura kinetic watches may also be used.

[0028] In various embodiments, a wristband may detect relative motion between itself and another device. For example, a player may wear two wristbands. One wristband may transmit a signal of a certain strength to the other wristband. Based on the distance between the wristbands, the signal may appear relatively strong (e.g., if the wristbands are close) or relatively weak (e.g., if the wristbands are farther apart) on the receiving wristband. In this manner, how close the wristbands are to each other may be measured. The relative motion of the wristbands may be measured relative to any suitable device. A player may wear a device in another location on their body, such as a belt buckle, that can transmit or receive signals. The wristband may transmit signals to or receive signals from a receiver attached to a wall, ceiling, floor, or any fixed device external to the person, such as a gaming device.

[0029] In various embodiments, the wristband may detect a drinking motion. The wristband may detect wrist rotation via an orientation sensor within the wristband. If there is a large wrist rotation, it may be inferred that the player is nearly finished with their drink and therefore needs to tilt it more. Accordingly, a casino representative may be instructed to offer the player a new drink and / or the player may be asked if they would like another drink.

[0030] (Technology for harvesting energy for wristbands) Various techniques for harvesting energy from the environment or from ambient conditions are described in the paper "Energy Scavenging for Mobile and Wireless Electronics" by Joseph A. Paradiso and Thad Starner. As of May 11, 2007, the paper was available at http: / / www.media.mit.edu / resenv / pubs / papers / 2005-02-E-HarvestingPervasivePprnt.pdf.

[0031] Radio frequency identification systems allow a tag to receive energy from a remote or non-proximate source (e.g., a tag reader) and receive the radio frequency energy from the tag reader inductively, capacitively, or radiatively.

[0032] Solar cells may enable portable devices such as wristbands to draw energy from ambient light. Exemplary technologies include crystalline silicon solar cells.

[0033] Thermoelectric generators may allow for the derivation of energy from heat transfer. These generators exploit temperature gradients, such as the difference between a person's body temperature and the temperature of the surrounding air. The Seiko Thermic watch uses a thermoelectric generator to power its mechanical watch components. One thermoelectric technology is Thermo Life from Applied Digital Solutions.

[0034] Various technologies enable energy harvesting from vibration or motion. The motion can be used to move a mass in a preferred or biased direction. The mass's movement can wind a spring. The spring's energy can then be used to generate direct mechanical energy (e.g., moving the hands of a watch) or to move magnets, coils, or other components of a generator to generate electricity. Exemplary technologies for harvesting energy from mechanical motion include the ETA Autoquartz, Seiko AGS (Automatic Generating System), and Ferro Solutions' Harvester. Piezoelectric materials can deform in the presence of motion or vibration to generate electricity. For example, Ocean Power Technologies has developed a harvester that is immersed in turbulent water and deforms with the water current to generate electricity. Some generators include capacitors with moving plates. On a charged capacitor, induced motion of one of the plates can generate a current. Piezoelectric and capacitive generators can be used to harvest energy from shoes, for example, while walking.

[0035] Some power generating devices include a turbine that can be driven by the ambient airflow.

[0036] (Gaming device as an antenna array) In various embodiments, two or more stationary gaming devices may each include antenna array components. When operating together, the gaming devices may detect and interpret signals from the mobile gaming device or from the wristband. For example, two or more stationary gaming devices may each have an antenna. The gaming devices may each receive a signal emitted by the mobile gaming device or the wristband. The signal received at each of the antennas of the two or more gaming devices may then be amplified, perhaps with some time delay or phase shift applied by one or more of the gaming devices. Increasing the signal received at two or more antennas may reduce the signal-to-noise ratio, potentially allowing the signal from the mobile gaming device or wristband to be read with greater accuracy or at greater distances, or may allow the mobile gaming device to allow the wristband to transmit with less power and therefore benefit from extended battery life.

[0037] (New battery at the end of every shift) In various embodiments, the battery or power source in the wristband may be replaced routinely. The battery may be replaced (a) once a day (e.g., at the end of the day); (b) once per shift (e.g., at the end of a casino participant's shift; e.g., at the beginning of a casino participant's shift); (c) once per hour; or any other basis. In various embodiments, the wristband may include an indicator light or some other output device to indicate a low power level of its battery or power source. The battery may be replaced or recharged when the indicator light is illuminated.

[0038] (The wristband gives the player location information.) In various embodiments, the wristband may broadcast a signal. This signal may include a player identifier, such as a name or card number that tracks the player. This signal may also include information about the player's location. For example, the wristband may gather location information from beacons or satellites, calculate its own location, and transmit the location information to the gaming device or any receiver.

[0039] In some embodiments, the wristband measures its change in position, but not its absolute position. A receiver receiving a signal from the wristband may be able to measure the wristband's orientation from the receiver, but not its distance. The player wearing the wristband may then walk some distance, and the position of the wristband may change accordingly. The wristband may include an accelerometer or other motion detector that can be used to measure the change in position (not necessarily the absolute position). The wristband may also include a sensor for measuring orientation, such as a compass. Thus, the wristband may measure the change in position (e.g., measured in feet or meters) and broadcast this change to the receiver. The wristband may also measure the direction in which the change in position occurred and broadcast this direction to the receiver. Again, the receiver may be able to measure the wristband's orientation from the receiver at the wristband's new position, but may not be able to measure its distance from the receiver. Based on these two measurements of the wristband's orientation from the receiver, based on the distance the wristband moved, and based on the direction the wristband moved, the wristband's absolute position may be determined. This is because the triangle formed by the receiver will know the initial position of the wristband, the final position of the wristband, one side and two adjacent angles. This side is the path the wristband traveled (assuming it took the shortest path), and these angles can be found based on the direction the receiver detected the wristband in its initial and final positions, and based on how the wristband itself moved.

[0040] (a wristband used to control a mobile gaming device) In various embodiments, the wristband may be used to control a mobile gaming device. The wristband may transmit signals to the mobile gaming device, where such signals provide instructions or commands regarding how to proceed in a game. Such instructions may include instructions to begin playing a game, to hold particular cards, to hit or stand (e.g., in blackjack), to bet on particular paylines, or any other instructions. The wristband may also transmit signals to a stationary gaming device, where such signals provide instructions or commands to the stationary gaming device regarding how to proceed in a game.

[0041] The wristband may measure its own motion via a motion sensor (e.g., via an accelerometer). The wristband may interpret such motion as commands to be used in a game. The wristband may transmit such commands to a mobile or stationary gaming device to control such a device. In some embodiments, the wristband may record motion data such as distance traveled, acceleration, trajectory, speed, or any other motion data. The motion data may be transmitted to the mobile or stationary gaming device. At the mobile or stationary gaming device, the motion may be translated into game commands. In various embodiments, the wristband may transmit either the motion data or the game commands to a casino server. The casino server may then transmit the motion data or the game commands to the mobile or stationary gaming device to control such device.

[0042] In various embodiments, the wristband may be used to control or issue commands to any device. Such devices may include a point-of-sale terminal, a vending machine, a kiosk, an automated teller machine (ATM), or any other device. For example, a player may create a series of motions with their hands. The motions may be received by the player's wristband. The wristband may interpret the motions as commands for the ATM. The wristband may then transmit the commands to the ATM. The ATM may then act on the commands, for example, by dispensing cash to the player.

[0043] (Wristband for 2D control) In various embodiments, a player may move their hand or arm in one plane. Such motion may cause a cursor on a screen to move in a similar manner. For example, if a player moves their hand first in one direction and then in the opposite direction, the cursor will also move first in one direction and then in the opposite direction. A player may rest their arm on a flat surface, such as a tabletop. A player may rotate their hand on the tabletop, thereby moving their hand in two dimensions. Thus, the wristband may be used to control the position of a cursor on a screen, such as the screen of a stationary gaming device, a mobile gaming device, or other device.

[0044] (The string provides force feedback) In various embodiments, a stationary game device may include a string, cable, wire, or other similar component. The string may be wrapped around a wheel, axle, spindle, shaft, or other device. The game device may include a motor to rotate the wheel. Rotating the wheel in one direction releases more string, while rotating the wheel in the other direction retracts the string.

[0045] In various embodiments, a player may attach the end of the string to a wristband. Depending on an event in the game, the gaming device may either pull the string in or release more of the string. This may have the effect of pulling and releasing the player's wrist. This may provide tactile feedback to the player. In some embodiments, a player may intentionally pull the string to produce a command in the game. For example, a player may pull the string outward to spin the reels of a slot machine game. The faster or harder the player pulls the string, the faster the reels may spin.

[0046] (Distinguishing signals from multiple wristbands) In various embodiments, a gaming device may detect a signal from a wristband. The wristband may transmit an identifier to the player, allowing the gaming device to recognize the player's identity. In various embodiments, when one gaming device detects a signal from a wristband, other gaming devices may also detect the same signal. Thus, in various embodiments, a gaming device may determine whether it was the player's intent to communicate with it or whether it was the player's intent to communicate with a different gaming device.

[0047] In various embodiments, a gaming device may recognize that someone is playing the gaming device. For example, the gaming device may detect actual button presses, the insertion of a player-tracking card, the insertion of currency, etc. At the same time, the gaming device may detect a signal from the wristband. The gaming device may display a message or otherwise ask the player currently playing the machine whether that player is the very person for whom the wristband signal was received. The gaming device may recognize the player's identity from the wristband signal and therefore display the player's name to the player physically present at the gaming device. Once the physically present player recognizes their own name, the player may confirm that the gaming device is, in fact, receiving the wristband signal from them. The gaming device may then allow the player to proceed with game play using motion controls.

[0048] In various embodiments, the gaming device may recognize that a wristband is nearby and that the gaming device is being played by a physically present player. Thus, a game may be conveniently started by, for example, physically depressing a button. The gaming device may then ask the physically present player whether they are the same player as indicated by the signal received from the wristband. If the physically present player answers in the affirmative, the gaming device may ask the player whether they wish to proceed using motion controls.

[0049] In various embodiments, a gaming device may distinguish between signals coming from different wristbands as follows: Each wristband may be associated with a unique identifier; Each wristband may broadcast its own unique identifier; The gaming device may query a physically present player as to which identifier corresponds to the player's wristband; In some embodiments, the gaming device may ask the player to enter the identifier of the player's wristband; if this identifier matches the identifier of a signal received from one of the wristbands, then the gaming device may only respond to signals received from that wristband.

[0050] In various embodiments, a gaming device may ask a player to bring their wristband near a reader. This reader may be an optical reader, an RFID reader, a magnetic stripe reader, or any other reader. In this way, the signal belonging to the player physically at the gaming device may be the strongest signal received by the gaming device. The gaming device may then allow the player physically at the gaming device to proceed with play using their wristband. The player may then use some motion control, or the player may use motion control for each command on the gaming device.

[0051] (Reference light for fixed game device) In various embodiments, a stationary gaming device may include one or more lights, beacons, transmitters, audio speakers, or other light-emitting devices. For example, a stationary gaming device may include two bright lights positioned on the gaming device. The light-emitting devices may serve as a reference point for a mobile gaming device and / or a wristband. The wristband may, for example, detect light or other signals from the two light-emitting devices on the gaming device. A bracelet may use the two light-emitting devices as a fixed reference frame to measure its orientation. For example, if the two light-emitting devices appear side-by-side from the wristband's perspective, the wristband may determine that its orientation is normal. However, if the two light-emitting devices appear one on top of the other, the wristband may assume that it has been rotated 90 degrees. In various embodiments, the light-emitting devices may output the same type of signal, e.g., light of the same wavelength and amplitude. In some embodiments, different light-emitting devices may output different signals. This may allow the wristband or mobile gaming device to distinguish one light-emitting device from another in all orientations, thereby determining its orientation even more accurately. In various embodiments, a stationary gaming device may have more than two light-emitting devices. For example, a stationary gaming device may have three, four, or five light-emitting devices. In various embodiments, the light-emitting devices may be located in other locations than directly on top of the stationary gaming device. For example, the light-emitting devices may be located on the ceiling or a wall.

[0052] In various embodiments, the light emitting device may emit light at a particular frequency. The light emitting device may emit red light, green light, infrared light, or some other frequency. The light emitting device may emit light at multiple frequencies. For example, the light emitting device may emit white light. The light emitting device may emit sound.

[0053] The wristband and / or the mobile gaming device may include a sensor, camera, microphone, or other detector to detect the output of the light emitting device. For example, the wristband may include a camera. The camera may detect light from the light emitting device on the gaming device. Based on the location of the light emitting device in an image captured by the wristband's camera, the wristband may determine its orientation.

[0054] In various embodiments, a gaming device may not necessarily have a dedicated light-emitting device for detection by a wristband or mobile gaming device. However, the wristband or mobile gaming device may detect gaming device-specific features. For example, the gaming device may have a candle on top that is intended to be lit when a casino player is called to the gaming device (e.g., when a player at the gaming device wins a jackpot). A sensor in the wristband or mobile gaming device may recognize an image of the candle. For example, the wristband may include a camera. The camera may capture an image and attempt to match portions of the image with pre-stored images of candles on the gaming device. The wristband may determine its orientation based on the orientation of the candle from the captured image relative to the orientation of the candle in a stored reference image. For example, if the captured image appears to be a 90° rotated version of the reference image, the wristband may assume that it has been rotated 90°.

[0055] In various embodiments, sensors in the mobile gaming device or wristband may detect other features of the stationary gaming device. The sensors may detect a pay table, a screen, a handle, a bet button, a coin tray, an image on the gaming device housing, a jackpot meter, or any other feature of the gaming device. For any feature, the wristband or mobile gaming device may store a reference image or signal. To detect or interpret a feature, the wristband or mobile gaming device may capture an image and attempt to match a portion of the image with one or more reference images. In the matching process, the wristband or mobile gaming device may manipulate the captured image, adjusting its size or orientation to attempt to better match the reference image. If there is a match (e.g., a portion of the captured image matches the reference image of a coin tray), the wristband or mobile gaming device may determine the degree to which the captured image needed to be rotated to achieve the match. This degree of rotation may then indicate the amount the wristband or mobile gaming device was rotated.

[0056] In various embodiments, the gaming device may track the motion of the wristband or mobile gaming device. The wristband may include a beacon or light-emitting device, such as an infrared light-emitting device, a light-emitting diode, or an audio speaker. The wristband may include two or more light-emitting devices. The gaming device may include a detector, such as a camera, a microphone, or an antenna. The gaming device may determine the position or relative position of the light-emitting devices on the wristband. For example, in a vertically upright position, two light-emitting devices on the wristband may appear side-by-side. When the wristband is rotated 90 degrees, one light-emitting device may appear on top of the other. Thus, based on the relative position of the two light-emitting devices on the wristband, the gaming device may be able to ascertain the orientation of the wristband. The apparent distance between the two light-emitting devices on the wristband may also provide an indication of the distance of the wristband itself from the gaming device. For example, if two light-emitting devices on the wristband appear close to each other, it may be assumed that the wristband is far away. On the other hand, if two light-emitting devices on the wristband appear far away from each other (at least relatively speaking), it may be assumed that the wristband is nearby. By tracking the motion of the wristband or mobile gaming device, a gaming device (e.g., a slot machine; e.g., a video poker machine) may ascertain commands intended by a player. The gaming device may execute those commands in the game it is playing. The gaming device may also transmit those commands to another device, e.g., another stationary gaming device, or e.g., a mobile gaming device.

[0057] (Screen navigation for motion control) In various embodiments, a gaming device, such as a stationary gaming device, may provide instructions to a player on how to use motion controls. The instructions may indicate one or more available commands that the player can give. For example, the gaming device may list commands for: (a) starting a game; (b) making a selection in a bonus round; (c) selecting a card to discard in a game of video poker; (d) selecting whether to hit or stand in a game of blackjack; (e) selecting a payline to bet on; or taking any other action in a game or otherwise. The gaming device may also provide instructions on how to issue a command. The gaming device may indicate which motion is required to issue the command. The gaming device may show a brief video or animation of people making a motion with their hands. Thus, a player may see a brief video clip of a person moving their arm in a particular way next to a potential command. This video clip may repeat continuously or may play on demand (e.g., upon contact by the player). The motion to be made to issue a command may be spelled out in text form, such as "Move your hand to the right twice and then up once." Instructions on how to use the motion controls may be presented in many different forms.

[0058] In some embodiments, a person may be guided by instructions and may have the opportunity to practice making a motion. For example, instructions for making a motion corresponding to a "start game" command may be played in the form of a video clip. In other words, an animation of a person making a particular motion may be shown on the display screen of the gaming device. The player may be instructed to repeat the motion on their wristband. The player may be instructed to follow a video of the motion being performed. If the gaming device recognizes the motion, the gaming device may subsequently ask the player to perform the motion for the next instruction. If the gaming device does not recognize a motion made by the player (e.g., if the player makes an incorrect motion), the gaming device may ask the player to repeat making the motion until the player has mastered the correct motion.

[0059] In various embodiments, when a player is playing a game on a gaming device (e.g., a slot machine) and the player makes a motion to issue a command, the gaming device may provide feedback as to how the gaming device interpreted the player's motion. For example, the gaming device may display the text message, "You have made a motion to start a new game."

[0060] (Time window for making motion) In various embodiments, there may be a finite time window during which a gaming device (e.g., a stationary gaming device) will accept motion commands. For example, there may be a 10-second window during which the gaming device will accept motion commands. During other times, a player may make motions, but they will not necessarily be registered as commands. This may give the player some freedom to make non-game-related motions (e.g., hand gestures in conversation) during times outside of the window during which commands may be registered. The time window for making motion commands may open and close periodically. For example, the window may open for 10 seconds, then close for 20 seconds, then open again for 10 seconds, etc. If a person makes their first motion command during the time window, this time window may be extended. For example, extending the time window may allow the person to complete an entire game before the window for making motion commands closes. In some embodiments, the time window for making motion commands may continue as long as the game is in progress. In some embodiments, the time window for making motion commands may continue for a predetermined time after the last motion command generated by the player. This may allow the player to continue making motion commands for as long as the player wishes. In some embodiments, there may be an alert or other indicator that the gaming device (e.g., a stationary gaming device; e.g., a mobile gaming device) accepts motion commands. For example, an indicator light on the gaming device may illuminate or change from one color to another. Thus, for example, the light may be blue when the gaming device accepts motion commands and red when the gaming device does not accept motion commands. In some embodiments, the player may turn motion control on or off. For example, the player may instruct the gaming device to enter a state in which it accepts motion commands or instruct the gaming device to ignore motion commands.A player may need to physically touch the gaming device to switch the motion commands either on or off. In some embodiments, when a gaming device does not accept a motion command, the gaming device may still respond to a motion command that instructs the gaming device to become accepting of other motion commands again. For example, the gaming device may then become accepting of motion commands again.

[0061] In various embodiments, the first set of motions may correspond to moving a cursor, mouse pointer, or other indicator. The second set of motions may correspond to making a selection. For example, if the cursor is resting over an image of a card or a button, making a motion from the second set of motions may correspond to selecting that card (e.g., selecting to discard that card) or pressing that button. A motion from the second set of motions may be used, for example, to select a bet amount, select a payline, select a decision from a menu of decisions, or make any other selection. A motion from the first set of motions may position the cursor for a later selection but may not yet commit the player to a course of action. In some embodiments, forward and backward (e.g., from the player's perspective) motions may correspond to the second set of motions, e.g., making a selection. Motions in other directions (e.g., up, down, left, right) may correspond to motions from the first set of motions, e.g., positioning the cursor.

[0062] In various embodiments, a player may receive visual feedback as they make a motion. A cursor may follow a path created by the player's wristband on the screen of a gaming device (e.g., a stationary gaming device; e.g., a mobile gaming device) as the player's hand moves. To make a particular command, a player may need to maintain the cursor within a particular boundary. For example, a boundary consisting of two concentric circles may be displayed on the display screen of the gaming device. A player may need to create a circle with the cursor while keeping the cursor outside the inner circle but inside the outer circle (i.e., between the two circles). In some embodiments, points or dots may be present on the screen. A player may need to make a motion such that the on-screen cursor is moved between two dots. In some embodiments, several pairs of dots may be present. A player must move the cursor in some particular order between various pairs of dots to issue a command. Different commands may require moving the cursor in different orders between different pairs of dots.

[0063] In various embodiments, a player may perform a motion command to position a cursor over a button. To select a button, the player may perform a further motion command. Various buttons may correspond to different commands or actions within the game. Thus, by performing a motion to position the cursor over the appropriate button, a player may perform a desired command within the game.

[0064] (The wristband senses muscle tension in the wrist in the form of a grasping motion.) In various embodiments, a player's wristband may include strain gauges. The wristband may be made of a flexible material, such as rubber. The wristband may fit snugly around the player's wrist. When a player closes their fist, the player may tense certain wrist muscles, which can place additional strain on the wristband as the circumference of the player's wrist increases. The strain gauges can sense this additional strain in the wristband. The strain gauges can send a signal to a processor in the wristband indicating the strain being detected. The strain gauges can also send a signal via an antenna or other transmitter to another device, such as a mobile gaming device, a stationary gaming device, or a casino server.

[0065] In various embodiments, the wristband may have one or more pressure sensors on an interior surface, e.g., the surface that contacts the player's wrist, that can sense pressure from the player's wrist, indicating possible strain in the wrist or flexion of the wrist muscles.

[0066] In various embodiments, the wristband may have temperature sensors that can detect increased temperature at the wrist due to increased blood flow and / or more rapid burning of energy in the wrist muscles. These sensor readings can correspond to tension in the player's own wrist, such as when the player performs a grasping motion.

[0067] In various embodiments, the electrical activity of nerves or muscles in the wrist may change depending on whether the muscles are tense or relaxed. A sensor in the wristband, such as an antenna, can sense the electrical activity in the wrist and interpret the electrical activity as an indication of whether the wrist muscles are tense or not.

[0068] In various embodiments, wrist muscle tension may be interpreted as a command in a game. In various embodiments, wrist muscle tension may be interpreted as selecting a button or selecting from multiple options. In various embodiments, wrist muscle tension may correspond to substantially grabbing something in a game. For example, in a bonus round, a game character may grab the knob of one of three doors to open it. Since wrist muscle tension can be triggered by a player actually making a grabbing motion (e.g., in the real world), the player can use the grabbing motion as an intuitive way to select or grab something in a game. Thus, for example, a player can move a cursor by linearly displacing their hand and select something by making a grabbing motion.

[0069] In various embodiments, a sensor or detector can detect grasping motions or other hand or wrist motions even if such sensors are not in a wristband. For example, a camera may capture the motions of a player's hands. Image processing algorithms may be used to recognize that the motions are being made by the player's hands. These motions may be translated into commands in the game.

[0070] Thad Starner, Joshua Weaver, and Alex Pentland of the Massachusetts Institute of Technology have developed a camera-based system for recognizing American sign language, which they describe in a paper titled "Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video."

[0071] (slot machine receiver) In various embodiments, a gaming device, such as a slot machine, may include a Bluetooth transceiver. The transceiver may be built into the device. The transceiver may also take the form of a Bluetooth dongle that can be connected to a Universal Serial Bus (USB) port on the gaming device. In various embodiments, the gaming device may include a Wi-Fi transceiver. The gaming device may send and receive messages to and from a wristband or mobile gaming device using Bluetooth, Wi-Fi, or any other communication protocol.

[0072] (Component of the message from the wristband) The data content of a signal from a wristband may include one or more components. It may be understood that the signal always includes those components in a particular order, for example. For example, the first 3 bits of the signal may indicate the start of a new message. The next 4 bits may indicate the type of device (e.g., wristband; e.g., mobile gaming device) providing the transmission. The next 30 bits may provide an identifier for the wristband. The next 100 bits of the signal may provide a player name. The next 20 bits may provide a command. The next 10 bits may indicate that the signal has ended. In some embodiments, the signal may include one or more of the following portions or regions: (a) a region indicating the start of the signal; (b) a region indicating the type of device sending the signal; (c) a region indicating the intended recipient of the signal (e.g., a unique identifier for the gaming device; e.g., an identifier for the casino server); (d) a region indicating a player identifier; (e) a region indicating a device identifier (e.g., a unique identifier for the particular device sending the signal); (f) a region indicating the end of the signal; (g) a region indicating a player name; (h) a region indicating a command to be used in the game; (i) a region indicating a name identifier (an identifier for the game to which the command applies); (j) a region containing one or more error checks; and any other regions.

[0073] Verifying Player Presence and Identification at Fixed Gaming Devices In various embodiments, the wristband can transmit a signal. The signal may be received by a stationary gaming device. The signal may include an identifier for the wristband. The gaming device can transmit the wristband's identifier to a casino server. The casino server can look up the name of the player signing into the wristband (e.g., the player currently using the wristband). The casino server can transmit the player's name to the gaming device. In some embodiments, the signal from the wristband may include a player identifier. The gaming device can transmit the player identifier to a casino server. The casino server can then transmit the player's name back to the gaming device. In any event, the gaming device can determine the player's name. The gaming device can display a message indicating the player's name. The message may be a greeting. For example, the message may say, "Hello, Sarah Jones!" The message may also ask the player to confirm his or her identity. A player can verify his or her identity by answering a secret question, providing biometric authentication (e.g., a fingerprint), inserting a player tracking card, inserting a credit card, inserting a bank card, inserting a driver's license, flashing any of the aforementioned cards in front of a camera, or any other manner. In various embodiments, a player can verify their identity through physical contact with a gaming device. For example, a player can answer a secret question by physically touching letters on a touchscreen of the gaming device and writing the answer that way. If a player verifies their identity through physical contact with the gaming device, the gaming device can be sufficiently assured that the gaming device cannot be controlled based on motion or other wireless commands from anyone other than the person sitting at the gaming device.

[0074] (Prominent screen for motion control only play) In various embodiments, a casino or other venue may include a large display screen. The screen may display a game. The screen may show the progress and action of a game, such as a slot machine game or a game of video poker. Electrical or other devices associated with the screen may enable the screen to receive motion inputs for playing the game. For example, there may be an antenna for receiving signals from a player's wristband or a camera for reading the player's motion commands. A processor or other device may calculate or determine game events or game outcomes. A player may provide value or currency for gambling by inserting a cashless game ticket. Accordingly, associated with the screen may be a ticket dispenser for accepting and dispensing cashless game slips.

[0075] A player can play a game on a large display screen. The player can issue commands in the game using motion controls. For example, the player's wristband can detect motions made by the player's hands. Motion instructions can be transmitted to the large display screen. The large display screen can then navigate the course of the game as directed by the player's commands.

[0076] In various embodiments, motion-controlled games using large display screens may be located at each end of two or more rows of slot machines. For example, at the end of each row of slot machines or other gaming devices, there may be a large display screen featuring a game using motion control. Such games may be visible to everyone in the row of slot machines. In this way, people playing the slot machines can see the games being played on the large screen and try out the motion control themselves.

[0077] (toggle button on wristwatch to activate or deactivate a function) In various embodiments, the wristband may include a switch, button, toggle, or other device for selecting between two or more states. The switch may be used to enable or disable motion control. Thus, when the switch is in one position, a player wearing the wristband can use motion control to control actions in the game. When the switch is in another position, the player cannot use motion control to control actions in the game. If a player does not want to play the game for a moment, the player can press the switch, disabling motion. The player could then make wrist gestures without worrying that the gestures would affect the outcome of the game. When the player wants to play the game again and use the game's motion controls, the player can press the switch to enable motion control once again.

[0078] In various embodiments, a player can use a switch or other device to activate or deactivate other features of the wristband. A player can activate or deactivate haptic feedback. For example, with a switch in one position, the wristband can provide force feedback or haptic feedback to the player. With the switch in another position, the wristband cannot provide such feedback. A player may wish to deactivate haptic feedback, for example, to conserve battery life in the wristband. In some embodiments, a player may turn sounds on or off. For example, in at least one state, the wristband may emit an audio signal. The audio signal may be related to the game (e.g., if the player wins, victory music may be emitted from the wristband). The audio signal may be related to the player's location. For example, if the player enters a restricted area where gaming is not allowed, the wristband may emit an audio signal. The audio signal may be related to account balance. For example, if a player's account balance reaches zero, the wristband may emit an audio signal. There may also be other reasons for an audio signal to be emitted by the wristband.

[0079] In various embodiments, the wristband may include one or more buttons, one or more sensors, one or more piezoelectric sensors, a batter, a transmitter, a receiver, and an onboard processor. The buttons allow the player to change the environment or state of the wristband (e.g., turn sound on or off). The buttons allow the player to give commands about the game, although such commands may not be motion-based. The sensors may include motion sensors such as accelerometers or gyroscopes. The sensors may include position sensors such as GPS sensors. The sensors may include temperature sensors, pressure sensors, strain gauges, microphones, light sensors, or any other sensors. The sensors can perform a variety of functions. The sensors can detect motion so that such motion can be translated into commands. The sensors can sense the player's location so that the player can be informed of whether the player is in an allowed gaming area. The sensors may be used to sense the player's muscle tension or electrical activity, for example, to drive motion commands. The transmitter may be used to communicate with another device, such as a fixed gaming device, a mobile gaming device, or a casino server. The receiver may receive a communication from another device, such as a mobile gaming device, a stationary gaming device, or a casino server. A communication received by the wristband may reprogram the wristband. Such a communication may, for example, provide a command to the wristband. For example, a communication received by the wristband may instruct the wristband to shut down due to a player's account balance reaching zero.

[0080] (holding hands) In various embodiments, the wristbands of two players may interact. The interaction may occur when the wristbands are in close proximity to each other. For example, if two players shake hands while wearing wristbands, the two wristbands may interact.

[0081] In various embodiments, during an interaction, a first player's wristband can receive information from a second player's wristband, and a second player's wristband can receive information from the first player's wristband.

[0082] In various embodiments, the mobile gaming device of the second player can receive information from the wristband of the first player. In various embodiments, the mobile gaming device of the first player can receive information from the wristband of the second player.

[0083] In various embodiments, a bet can be made or determined by shaking hands between two players. Technically, in some embodiments, the bet can be made when the wristbands of two players are within a predetermined distance (e.g., 5 inches (12.7 cm)) of each other for a predetermined time (e.g., 5 seconds). In some embodiments, the bet can be made when the wristbands are within a predetermined distance of each other for a predetermined time and when there is a shaking motion on one or both of the wristbands. The shaking motion corresponds to shaking hands. The wristbands may communicate with each other about the timing of the shaking motion to ensure that the wristbands are shaking simultaneously when shaking hands. In various embodiments, a first player may predetermine the time of the bet using a fixed gaming device or other device. For example, a first player can place a bet so that the first player wins $1 from a second player if the roulette wheel spins on black, but the second player wins $1 from the first player if the roulette wheel spins on red. Once a bet is specified, the first player only needs to find a second player to shake hands with to finalize the bet. In various embodiments, the first player may mischaracterize the duration of the bet to the second player. Thus, in various embodiments, the first player may be allowed to predetermine only fair bets (e.g., bets where both sides have an equal chance of winning and / or bets where both sides have an equal expected win and / or bets where both sides are expected to win and lose zero). In various embodiments, when players shake hands to place a bet, the duration of the bet may be displayed on one or both of the players' mobile gaming devices. Each player may have a time window (e.g., 30 seconds) to cancel the bet. To cancel the bet, the player may, for example, press a "cancel" button on their mobile gaming device.If no player cancels their bet, an outcome can be made and the bet can be decided one way or the other.

[0084] In various embodiments, a first wristband can detect the proximity of another wristband. The wristband may be Bluetooth so that the wristband can detect the proximity of another wristband transmitting over the Bluetooth protocol. In various embodiments, the wristband may be programmed or configured to send and receive signals over other protocols, such as Wi-Fi.

[0085] In various embodiments, two or more players may join hands to place a bet with each other. The player who wins may depend on the outcome of several games, such as games played or simulated by the gaming devices. In some embodiments, two players must be in close proximity to a gaming device, such as a fixed gaming device, to determine the bet. For example, two players must be standing in front of a slot machine to make the bet. Players may be required to be within a predetermined distance of a particular gaming device, such as within two feet. Wristbands of one or two players can communicate with the gaming device indicating that the players have agreed to the bet. One or both wristbands can communicate with the gaming device the duration of the bet, such as the game on which the bet depends. The gaming device can then play the appropriate game to satisfy the bet. For example, if the bet is for a game of video poker, the gaming device can play a game of video poker. If the bet is for a game of blackjack, the gaming device can play a game of blackjack. In various embodiments, the wristbands can communicate with the gaming device if a player wins under either circumstance. For example, the wristband can communicate with the gaming device that if the house wins the game of blackjack, "Joe Smith" wins, while if the player wins the game of blackjack, "Jane Smith" wins. In this case, the player may be said to be a virtual player simulated by the gaming device. The gaming device can play a basic strategy or an optimal strategy on behalf of the virtual player. In some embodiments, two players who bet on a game can play the game against each other using one or more gaming devices. The players can direct strategy decisions at the gaming devices. For example, if two players bet on a game of blackjack, the players can effectively agree to play a game of blackjack against each other. The two players can play on specific gaming devices. During the course of the game, the players can make decisions about the game.A player can make decisions by physically pressing buttons on the gaming device or by physically interacting with the gaming device. A player can also make decisions by using motion controls, for example, using the player's wristband.

[0086] (Hand-holding incentives) In various embodiments, there may be an incentive to shake hands with people. A person's wristband may track the number of times the person has shaken hands with someone else and / or the number of people with whom the person has shaken hands. In some embodiments, after shaking hands with each other, the player's wristband may transmit a record of the handshake or other indication to the casino server. The wristband may transmit an identifier for the other player or other wristband the player has come into contact with. The casino server and / or the player's wristband may track the number of times the player has shaken hands with other players. The casino server and / or the player's wristband may also track the names or identification of other players with whom the player has shaken hands. In various embodiments, the player who shakes hands with the most other players in some period of time (e.g., a day) may win a prize, such as $1,000.

[0087] In some embodiments, a mixer may be held at a casino or related property or any other venue. The mixer may be an opportunity for singles to meet, an opportunity for business people to connect, an opportunity for scientists to exchange ideas with colleagues, or any other type of mixer. During the mixer, people may shake hands. People's wristbands may automatically exchange information including names, contact information, email addresses, phone numbers, biographical information, photos, credentials, place of residence, age, gender, marital status, or any other information that may be appropriate for the situation.

[0088] The wristbands of people attending a mixer can transmit to a casino server or other device information about the people they shook hands with or made contact with. People who were at the mixer can later log in to a website to see a summary list of the people they met. The website may also include contact information about people. In some embodiments, contact information is not provided. Rather, a person must select the people they want to contact. If a person selects another person, and the other person selects him / her, the website can later provide them both with each other's contact information.

[0089] In some embodiments, during a handshake, a person's wristband can transmit information about that person (e.g., contact information) to another person's mobile device (e.g., a mobile gaming device; e.g., a personal digital assistant; e.g., a mobile phone). In this way, at the mixer's end, a person can save information on their mobile device about other people they met during the mixer.

[0090] In various embodiments, at the mixer end, a person can view his / her images of the people he / she met at the mixer. Viewing the images can jog the person's memory about the people he / she met. The person can select people he / she is interested in contacting further. The person can then be provided with their contact information. In some embodiments, the person can only be provided with their contact information if they are also interested in seeking further contact with the person.

[0091] In various embodiments, the mixer is held in a bar, restaurant, lounge, gym, swimming pool, gambling floor, or any other lounge.

[0092] (Pay by shaking hands) In various embodiments, players can pay by shaking hands. Players can pay for drinks, foot items, retail products, or any other items by shaking hands. In some embodiments, casino employees or retail employees may possess wristbands. When the employee shakes hands with a person (e.g., a customer; e.g., a player), the employee's wristband can receive a communication from the player's wristband. The communication may include information about the player, such as a name, identifier, credit card identifier, financial account identifier, or any other information about the player. The employee's wristband can communicate the player's financial account identifier and other identifying information about the player to a point-of-sale terminal, a retail server, a casino server, or any other device. The player can then charge the purchase via a credit card network or other financial network.

[0093] By shaking hands with a casino employee, retail employee, salesperson, or other person, the player may have a limited time to evaluate the transaction and cancel it. For example, the player's wristband may also store the transaction details after shaking hands with the salesperson. The transaction details may include the purchase price, product, delivery method, etc. The player can bring their wristband close to a mobile or stationary gaming device. The wristband can transfer the transaction details to the mobile or stationary gaming device. The mobile or stationary gaming device can then display the transaction details for the player. The player can evaluate them and decide whether to cancel. If the player wants to cancel, in some embodiments, the player can press a button or screen area on the mobile or stationary gaming device. The player may also be required to return to the location where the player purchased the product and return the product.

[0094] In various embodiments, a player may hold their wristband close to a reader as a way to pay for a transaction. The player may touch the pad with the wristband. For example, a player may place their hand on the pad to pay for a drink. The pad may include an antenna or other type of receiver to detect a signal from the wristband. The detected signal may include a financial account identifier.

[0095] In various embodiments, a player can use their gaming credit balance to pay for purchases or other commercial transactions. The player may have an account of gaming credits that is stored and tracked on a casino server. When a player holds their wristband near a pad or reader to make a purchase, the reader can verify with the casino server whether the player has sufficient account balance to complete the purchase. In various embodiments, the pad or reader may provide a first indication if the player has sufficient account balance and a second indication if the player does not have sufficient account balance. The first indication may be, for example, a green light. The second indication may be, for example, a red light.

[0096] (The wristband will be unclasped) In various embodiments, if the wristband is separated from the player (e.g., if the wristband becomes unclasped), an alert may be sent to the casino server. The alert may indicate to the casino server that the wristband is no longer encircling the player's wrist. In various embodiments, once the wristband is removed, the wristband may cease to function for gaming purposes. For example, the wristband may no longer provide motion control. The wristband may also cease to communicate a player identifier to the mobile gaming device. Thus, the player's mobile gaming device may no longer allow the player to engage in gambling activities. Various other functions of the wristband may also cease once the wristband is removed.

[0097] In various embodiments, if a player wishes to restore various functions of the wristband, the player may visit a special service area of ​​the casino, such as a casino desk. There, a casino employee may return the wristband to the player. The casino employee may transmit a special code to the wristband to reactivate it. The casino employee may also check the player's identity by, for example, requesting a fingerprint or driver's license before reapplying the wristband.

[0098] In various embodiments, the wristband includes one or more sensors for determining whether the wristband has been released from the player, unclasped, or otherwise tampered with or removed. For example, the sensors may include an electrical circuit surrounding the wristband. If the wristband is released, the circuit may be disrupted.

[0099] In various embodiments, the wristband or mobile gaming device may rely on continuous or periodic contact with the casino server to function. If the wristband or mobile gaming device loses contact with the casino server, it may cease functioning. In various embodiments, the wristband may periodically communicate with the server. Input the wristband receives from the player may not be executed until the next communication is received from the server. For example, if a player moves his or her hand to create a command, the wristband may save a record of the motion and / or save a command corresponding to the motion. However, the wristband may not transmit that command to another device, such as the mobile gaming device or gaming device the player is playing on. Rather, the wristband may save that command until it again receives a communication signal from the server. In this way, the wristband may ensure that a command or game command is not executed while the wristband is unable to contact the casino server. In some embodiments, the wristband may store input received from the player. However, if the wristband does not receive communication from the casino server within a predetermined period of time to receive the input, the wristband may discard the input. In this way, a player cannot be surprised later when multiple save or saved commands are executed simultaneously. In various embodiments, a player entering an elevator may be unable to play for a while because communication between his bracelet and the casino server may be lost.

[0100] In various embodiments, instead of a wristband ceasing to function when opened or unclasped, the wristband could continue to broadcast "I'm open" to the server until the server confirms. There could be a period of time after the wristband is opened for it to attempt to tell the server that it's open. Then, there could be a period of time after it receives confirmation from the server before it stops broadcasting. After the wristband is opened, it may no longer allow some functions (e.g., payments to be made with the wristband), but may still allow other functions (e.g., motion control). That is, in various embodiments, some functions will not function upon release of the clasp or otherwise removal of the wristband.

[0101] (The wristband and the mobile gaming device can replicate each other's functions.) In various embodiments, any motion command that can be made at a wristband can also be made at a mobile gaming device. For example, just as a wristband can include sensors that detect acceleration, changes in direction, displacement, and any other motion, so too can a mobile gaming device. Like a wristband, a mobile gaming device can include a processor for reading signals from motion sensors on the mobile gaming device and interpreting such motion as a command to be used in a game or as any other command. In various embodiments, any command that can be made via a mobile gaming device can also be made using a wrist. In various embodiments, a wristband can detect a motion made by a player and send an indication of the motion to the mobile gaming device. The mobile gaming device can interpret the motion as a command in a game or as any other command. In various embodiments, a mobile gaming device can detect a motion and send the motion to a wristband. The wristband can interpret the motion as a command in a game, for example. The wristband can then send an indication of the command to a stationary gaming device. In various embodiments, any signals or alerts broadcast by the mobile gaming device based on the location of the mobile gaming device may be similar to those broadcast by the wristband based on the location of the wristband. For example, if a player strays outside of a legal gaming area, the mobile gaming device or wristband may detect the player's location and issue an audio alert for the player. In various embodiments, any tactile feedback that may be provided by a wristband may also be provided by the mobile gaming device. In various embodiments, any tactile feedback that may be provided by a mobile gaming device may also be provided by the wristband. In various embodiments, any information received, determined, or detected by the wristband may be communicated to the mobile gaming device, for example, via wireless communication.

[0102] The following are embodiments, not claims. Various embodiments include: A. A method comprising: receiving a first wireless signal from a first device; receiving a second wireless signal from a second device; determining a first player identifier from the first wireless signal; determining a second player identifier from the second wireless signal; Displaying a message asking the player to identify himself; receiving an indication of a third player identifier via tactile input; determining that the third player identifier matches the first player identifier; receiving a third wireless signal from the first device; interpreting the third wireless signal as a command in a gaming game; and Executing commands in a gambling game. Executing a command may include fulfilling a command, following a command, acting in response to a command, and / or acting in accordance with a command. B. The method of embodiment A, wherein the first device is one of: (a) a wristband; (b) a watch; (c) a bracelet; (d) an armband; and (e) a mobile gaming device. C. The method of embodiment A, wherein determining the first player identifier from the first wireless signal includes determining the name of the first player from the first wireless signal. For example, the first wireless signal may encode a player name. In some embodiments, the player name may be found from a database that associates players with other player identifiers (e.g., player tracking card numbers) with player names. D. The method of embodiment A, in which the first player identifier and the second player identifier correspond to different players. E. The method of embodiment A, wherein receiving an indication of the third player identifier via tactile input includes receiving an indication of the third player identifier being entered using a button. For example, someone may enter the third player identifier by physically pressing a button (e.g., a letter key) on the gaming device. F. The method of embodiment A, wherein receiving an indication of a third player identifier via tactile input includes receiving an indication of the third player identifier being input using a joystick. G. The method of embodiment A, wherein receiving an indication of a third player identifier via tactile input includes receiving an indication of the third player identifier being entered using a touchscreen. H. The method of embodiment A, wherein receiving an indication of a third player identifier via tactile input includes receiving an indication of the third player identifier being input using a trackball. I. The method of embodiment A, wherein the third wireless signal encodes a set of motions created by the first device. For example, the third wireless signal may include a set of numbers representing position, velocity, acceleration, displacement, angular displacement, or other components of motion. The numbers may be understood to represent degrees, centimeters, or other units of measurement. In some embodiments, the third wireless signal may include an identifier for one of the set of recognized motions (e.g., "motion F"; e.g., "zigzag motion"). J. The method of embodiment A in which interpreting the third wireless signal includes interpreting the third wireless signal as a command to discard a card in a game of video poker. K. The method of embodiment A, wherein interpreting the third wireless signal includes interpreting the third wireless signal as a command to initiate a slot machine game. L. Equipment, including: Circularly formed bands; Band-mounted power supply; motion sensors attached to the band; an electromagnetic transmitter attached to the band; band-mounted audio speakers; tactile transducers attached to the band; a processor attached to the band; and An electromagnetic receiver attached to the band. The band can be a metal band, a rubber band, a chain band, a fabric band, a leather band, or any other type of band. In some embodiments, the band can be made into a loop by fastening its two ends together. In some embodiments, the band will always remain in a looped form except for unintentional ruptures or tears. M. The device of embodiment L, wherein the tactile transducer is operable to generate vibrations in response to an electrical signal from the processor. For example, the processor may instruct the tactile transducer to vibrate when a jackpot is won in a game being played by the wearer of the device. N. The apparatus of embodiment L, wherein the motion sensor is an accelerometer. O. The apparatus of embodiment L in which the processor is operable to: receiving a first electrical signal from a motion sensor; determining a first command for the first gambling game based on the first electrical signal; transmitting a first command to the electromagnetic transmitter; The electromagnetic transmitter is instructed to transmit a first command to the first gaming device. Thus, in various embodiments, a device may detect a player's motions and interpret the motions as commands in a gambling game, such as a slot machine game, a video poker game, a blackjack game, or any other game, and then transmit the commands to a gaming device, such as a slot machine or mobile gaming device, so that the commands can be executed in the game. P. The apparatus of embodiment L in which the processor is operable to: receiving instructions from the electromagnetic receiver that are wirelessly received by the electromagnetic receiver; receiving a second electrical signal from the motion sensor; following instructions to determine a second command for a second wagering game based on the second electrical signal; transmitting a second command to the electromagnetic transmitter; and The electromagnetic transmitter is instructed to transmit a second command to the gaming device. Q. It also includes a switch attached to the band. The apparatus of embodiment L, wherein the switch has two stable positions, and the processor is operable to detect the position of the switch and to direct the electromagnetic transmitter to transmit a signal only when the switch is in a first of the two stable positions. In various embodiments, a player may turn some or all aspects of the wristband on or off. The player may do this using a switch, button, or other switching device or other device. In one state of the switch, the wristband may transmit a motion or command to be used in the game. In another state of the switch, such motion or command may not be transmitted. For example, a player may want to create a motion without worrying that such motion will be counted in the game. R. The apparatus of embodiment L, further comprising a piezoelectric sensor attached to the band. The piezoelectric sensor may detect flexion of the player's wrist muscles, for example, via the pressure the muscles exert on the wristband. S. Equipment including: a housing with an upper surface parallel to the ground; Coin hopper located inside the housing; a bill validator mounted in the housing; a display screen attached to the housing; a processor located within the housing; a radio receiver mounted in the housing; a radio transmitter mounted in the housing; a first light source mounted on the top surface of the housing, operable to emit light at a first frequency; and a second light source mounted on the top surface of the housing at least one foot away from the first light source, the second light source configured to emit light at a second frequency different from the first frequency; The device may represent a gaming device. The two light sources may provide fixed reference points relative to which the wristband or mobile gaming device may determine its own position or orientation. For example, the first light source may be a green light and the second light source may be a red light. The wristband may detect the two light sources, for example, by capturing an image including the light sources, determining the apparent distance of the light sources in the image, and determining its own distance from the light sources based on the known distance between the two light sources. T. The apparatus of embodiment S in which the processor is operable to: Playing gambling games; and The course of the gambling game is altered based on the wireless signal received by the wireless receiver. In various embodiments, altering the course of a gambling game may include taking one of two or more possible actions in the gambling game, such as selecting one or more possible cards to keep, or selecting one of two or more possible bets.

[0103] (Some tactile technologies) The Impulse stick, manufactured by Immersion, is a joystick that provides force feedback and is marketed for use in rugged environments such as arcades.

[0104] The VibeTonz® system from Immersion is a system that can provide haptic sensations to a mobile phone. Such sensations can provide the feel of a replicating machine gun, the impact and decay of an explosion, or the feel of a foot striking a ball.

[0105] A "haptic interface device" provides haptic sensations (tactile displays) to a user of the haptic interface device in response to the user's interaction with an environment with which the haptic interface device is associated. "Haptic" refers to the sense of touch: thus, a haptic interface display device generates sensations of touch, such as sensations associated with texture, force (e.g., friction, magnetic repulsion, or magnetic attraction), vibration, mass, density, viscosity, temperature, humidity, or some combination of such sensations. Haptic interface devices can be embodied in a wide variety of devices, including, for example, devices for transmitting force and / or vibratory haptic sensations (e.g., styluses, movable arms, wheels, dials, rollers, sliders, or vibrating surfaces), devices for transmitting thermal sensations (e.g., thermally controlled surfaces or airflow), and devices for transmitting humidity sensations (e.g., humidity-controlled surfaces or airflow). Haptic interface devices can be used in a wide variety of applications. For example, some joysticks and mice used with computers incorporate force feedback to provide tactile indications to users of the joystick and mouse. Some paging devices are adapted to vibrate when a paging signal is received. Some toys generate vibrations as part of the toy's interaction. These examples suggest the breadth of applications in which haptic interface devices may be used.

[0106] In conventional haptic interface devices, the characteristics of the haptic display experienced by the user are determined by a haptic model that relates the state of one or more aspects of the environment to haptic sensations imparted to the user. The user interacts with the environment through the environment interaction model (directly or via the haptic model) using an environment interaction control device. The haptic model "interprets" the user's interaction with the environment (based on information about the user interaction obtained from either the environment interaction model or the environment) and generates a corresponding haptic display on the haptic display device. The environment interaction model can also cause a non-haptic display device to generate a non-haptic display (e.g., a visual and / or audio display). However, a non-haptic display is not necessarily required.

[0107] The magnitude of change in haptic sensation per unit change in the state of one or more aspects of the environment is referred to herein as the "resolution" of the haptic display. For example, in a haptic interface device used for video browsing and / or video editing, a knob can be rotated to advance frames in a video recording, and a force is applied opposite the knob rotation to simulate detents at predetermined transitions in the video recording from one video frame to the next. The resolution of the haptic display in a haptic interface device can be the frequency of occurrence of detents in the video recording (e.g., the number of video frames between each detent). (As illustrated by examples discussed further below, it is also possible to specify the resolution of the haptic display of such a haptic interface device in terms of the frequency of detents per unit duration over which the video was acquired.)

[0108] The output generated by the tactile display device may include, for example, sensations of texture, force (e.g., friction, magnetic repulsion, or magnetic attraction), vibration, mass, density, viscosity, temperature, humidity, or some combination of such sensations. When the environment is a visual and / or audio recording, for example, forces can be applied in opposition to the movement of the device embodying the environmental interaction control device, and a tactile display device simulating detents as a transition is created from one video frame (or other related set of visual recording data) to the next. Furthermore, the tactile model can reproduce various characteristics of tactile sensations, such as inertia, damping, and / or compliance. The tactile display device can utilize various devices to generate the tactile display. For example, if appropriate for the desired tactile display, devices for generating force and / or vibration tactile sensations can be used, including, for example, DC servo motors, voice coil motors, linear actuators, hydraulic actuators, pneumatic actuators, shape memory alloys (SMAs), and piezoelectric transducers. Thermal devices can additionally or alternatively be used where appropriate for the desired tactile indication, including, for example, thermoelectric modules or heater and fan combinations. Humidity devices and / or materials can additionally or alternatively be used where appropriate for the desired tactile indication, including, for example, capacitors, foggers, moisture-permeable barriers, and water-free materials.

[0109] The tactile display device may be embodied, for example, by a force-activated wheel, knob, handle, or arm, a heat source and / or heat dissipation device, or a humidifier and / or moisture absorber.

[0110] Various devices actively respond to user input by providing tactile cues or responses to the user. A cell phone vibrator or pager is a good example. Other examples include input keys that provide a clicking sound when moved; keys or touchscreens that suddenly move or vibrate in the direction opposite to the input; and keys that suddenly move or vibrate perpendicular to the direction of the input in response to a transducer attached to the device housing.

[0111] The display and / or input mechanism, such as a key, may be configured to provide active tactile force feedback. An electromechanical transducer, such as a voice coil-based linear vibration motor, a piezoelectric actuator, or a piezoelectric vibrator, is mechanically connected directly to the display, and an electromechanical transducer, such as a vibrator, is mechanically connected directly to the key.

[0112] In various embodiments, the haptic interface module is configured to output a pulse of a predetermined or user-defined amplitude and duration in response to receiving a trigger signal from the telephone processor. Alternatively, other interface logic (e.g., address decoding logic) is included between the digital signal bus and the haptic interface module. The telephone processor is programmed to trigger the haptic interface module in response to a predetermined condition as determined by intelligent operation within the telephone processor. Optionally, triggering the haptic interface module can be selectively enabled or disabled according to user-editable configuration settings. The haptic interface module is connected to an electromechanical transducer. The electromechanical transducer is driven by the output of the haptic interface module.

[0113] More generally, the electromechanical transducer is preferably driven by a signal including at least one approximation of a step function. (Note that a step function is a mathematical ideal that real-world circuits cannot achieve.) The step function includes a wide range of frequencies. By using a drive signal including an approximation of a step function, the electromechanical transducer is caused to emit an impulse of mechanical energy that propagates to a tactile point and is felt by a user operating a mobile phone. In various embodiments, the electromechanical transducer is driven by a signal including one or more pulses. A pulse, e.g., a single pulse or a complex waveform, is generated in response to each detected state, where a state represents a particular situation identified by the phone processor. Using a known pulse is advantageous in that the known pulse generates an impulse of mechanical energy that creates a tactile sensation that simulates the sensation of a previous state that the user may be familiar with.

[0114] The transceiver module, telephone processor, A / D, input decoder, D / A 510, tactile interface module, display driver, memory, and display driver are preferably part of an electrical circuit embodied in circuit components and interconnected with traces on a circuit board.

[0115] Alternatively, instead of using the phone processor, a different electrical circuit can be used to drive the electromechanical transducer to generate tactile feedback at the tactile point.

[0116] The haptic interface module may alternatively be a pulse generator that generates digital pulses of various widths, heights, and / or frequencies based on instructions from the telephone processor. An amplifier may be required depending on the impedance matching and current sourcing / sinking capabilities for the electromechanical transducer. Alternatively, the haptic interface module may simply be a current amplifier, and the pulses may be generated by the telephone processor itself. Another possibility is that the haptic interface module may include multiple DACs that apply analog signals for cases where additional audio channels are included.

[0117] Various situations can prompt different tactile responses. For example, on a pager or cell phone, a message or call from a spouse might cause all tactile points to vibrate, or a message or call from a boss might cause tactile points to vibrate in a circle around the electronic device, or a message or call from another person might cause tactile points to vibrate repeatedly on one side of the electronic device. The use of multiple adjacent vibrators in succession as described creates the illusion of movement (known as a rabbit on the skin).

[0118] This illusion of motion can be used to provide directional information for movement. Movement along one side, around the electronic device, back and forth can also be used to convey information such as attention-grabbing information, emphasis information, and general non-verbal information. The electronic device can also relay information about its status, such as out of range, low battery, and busy signals. Such information can be useful while the user is holding the electronic device to his / her ear and cannot easily see the information on the screen.

[0119] Multi-localized force feedback can also be used for sensory communication. Instead of sending voice or text messages, or picture or data files, specific haptic patterns can be sent to other users. The patterns can represent reminders, specific moods (e.g., thinking of you, love you, miss you, etc.), specific emotions, or any other user-defined content.

[0120] Computing devices are widely used for entertainment activities such as playing games. Currently, popular gaming computing devices include game consoles connected to home televisions, such as the Nintendo 64 manufactured by Nintendo Corp., the Playstation manufactured by Sony Corp., and the Dreamcast manufactured by Sega Corp. Gaming computing devices also include personal computers, such as Windows PCs and Macintosh computers. Portable computing devices are also often used for entertainment purposes, such as the Game Boy manufactured by Nintendo, personal digital assistants, such as the PalmPilot manufactured by Palm Computing, and laptop computers.

[0121] Users of these computing devices typically interact with games or other application programs using interface devices connected to a host computer (e.g., a gaming console). Such interface devices may include joysticks, gamepads, mice, trackballs, styluses, steering wheels, or other devices. Users move user-manipulable objects (manipulandums) such as joysticks, wheels, mice, buttons, dials, or other objects, which are sensed by the host computer and used to manipulate the graphical environment displayed by the host computer. Recently, haptic feedback in interface devices has also become available, where microprocessors on the host computer and / or interface device control one or more motors that output forces to the user. To further immerse the user in the gaming experience or interface task, these forces are associated with events or objects in the graphical environment. Here, the term "haptic feedback" is intended to include both tactile (or vibrotactile) feedback (forces delivered to the user's skin surface) and kinesthetic (forces provided in the manipulandum's degrees of freedom of movement) feedback.

[0122] Current force-feedback "gamepad" controllers (or add-on hardware for gamepad controllers) used to interface with games running on game consoles include the Dual Shock™ from Sony Corp., the Rumble Pak™ from Nintendo Corp., and the Jump Pack from Sega Corp., as well as other types of handheld controllers such as the MadCatz Dual Force Racing Wheel. These devices are inertial haptic feedback controllers that utilize one or more motors to vibrate the controller's housing, thus providing an output force, such as a vibration, to the user that is associated with game events and interactions. Typically, an eccentric rotating mass (ERM) motor, or pager motor, is used to generate the vibrations on the controller and, therefore, to the user. The motor is rigidly connected to the controller housing, providing a mass to the rotating shaft offset from the axial rotation so that when the shaft is rotated, inertial forces from the moving mass cause the motor and gamepad housing to rock back and forth.

[0123] To recreate textures, a force feedback device is preferably used to allow the user to touch and feel computer-generated objects. The sensation of touch is preferably simulated using a haptic (sensation / touch) interface. A haptic interface is a force-reflecting device that allows the user to touch, feel, manipulate, create, and / or modify simulated three-dimensional objects in a virtual environment. Various known haptic interface objects exist, including planar area interfaces, joysticks, gloves, thimbles, sticks or pens, exoskeletons, treadmills, fans, and magnets. Utilized hardware includes DC brushless motors, potentiometers, a Silicon Graphics, Inc. IRIS Indigo computer, a V25 board computer, an 8086-compatible microprocessor, a CRT display, a stereoscopic imager, magnetic and electromagnetic components, pulleys, a steel belt drive train, a VME bus, decoders, potentiometers, motor controllers, decoders, and cable reducers. The software required can be any of a variety of programming languages ​​(eg, C, C++) that can interface with a visual modeling program.

[0124] Currently, there is no consensus among experts regarding the "best" type of interface. However, a well-known example of a haptic interface is the "Phantom Haptic Interface" developed at MIT's Artificial Intelligence Laboratory. The "Phantom Haptic Interface" delivers precise tactile stimuli to humans with a previously unattainable level of fidelity and convenience. The device is designed to deliver forces that create "point contact," giving the sensation of fingertips interacting with a wide variety of objects. To achieve this, only three motors and three sensors are required, providing a computationally and mechanically tractable method for enabling haptic interaction with complex virtual objects.

[0125] Haptic interfaces allow users to touch and manipulate virtual, computer-created objects in a way that evokes "real" tactile sensations. This technology allows a user at a computer terminal to touch objects that exist only in the computer's "brain." By sending the correct digital signals to a master haptic interface device at a remote user location, the master device can be used to make the user feel as if they are performing a real task. In reality, the user may simply be interacting through motors equipped with a computer program.

[0126] Various embodiments are optically based and generally use unobtrusive, specialized data that is located on or embedded within the object whose 3D position and / or orientation is desired to be input into a computer. Typically, such data is viewed with a single TV camera or two TV cameras forming a stereo pair. The camera position may be near a computer display device looking outward, or near a human work or play area.

[0127] Beads such as those made from retroreflective glass bead tape or Scotchlite 7615 manufactured by 3M Co. provide data in the form of points, lines, or other desired shapes, which can be easily attached to any desired object, and which, when illuminated with incident light along the optical axis of vision, such as that of a TV camera, have high brightness and contrast with surroundings such as people, fabrics, rooms, etc. This also allows the camera to be used in normal environments and have fast integration times to capture the general motion desired, and allows the data to be easily distinguished, which greatly reduces computer processing time and processing costs.

[0128] Figure 14a 14a illustrates an exemplary single camera according to an embodiment. In this case, a user C5 points to an object C6 represented electronically on a screen C7 and wishes to have his pointing action registered with software contained in a computer C8 for that object (a virtual object) in order to cause a display device C7 to generate a signal to activate the object or move it (e.g., by subsequent finger motion or otherwise). He accomplishes this using a single TV camera C10, typically positioned above the screen as shown or positioned to the side (as in C11), to determine the position of his fingertip C12 in space and / or the pointing direction C13 of his finger.

[0129] It may also be desirable to use retroreflective material on the finger (e.g., either temporarily attached to the finger as jewelry, or painted onto the finger using a retroreflective coating "nail polish," or stuck to the finger with something like adhesive tape having a retroreflective coating). Such coatings include Scotch-lite 7615 and equivalents, which have specific reflectivity for easy identification and good contrast with their surroundings. The brightness of the reflection allows for dynamic target acquisition and tracking at the lowest cost.

[0130] The use of retro-reflective and / or highly distinctive targets (e.g., glowing orange triangles) allows for reliable target capture in general situations and does not limit the device to pointing at desktop applications under controlled lighting. Active (self-illuminating) targets such as LEDs may also enable such capture.

[0131] If we consider a camera system C10 that is above the screen C7 and that looks at the user, or more specifically, the user's hands, there is a relatively large field of view, so that the user's face can also be seen, as is typical for Internet telephony. This same field of view can be used in various embodiments, but it describes a relatively large amount. For greater precision, an add-on lens or zoom lens on the camera can be used to increase the resolution.

[0132] Or, according to various embodiments, it is possible to have multiple cameras, one for the Internet and others for the input applications described herein. Indeed, with ever-falling prices, the price of actual cameras with plastic lenses on CMOS chips is so low that it would probably be possible to have many cameras with fixed magnification, each with a separate chip!

[0133] These can be easily daisy-chained with either Fire Wire or USB so that they can actually be selected electronically either by different magnifications or by pointing in the desired direction.

[0134] Returning to the problem of determining the position or orientation of a human part, typically a hand or finger, in this case a finger. In various embodiments, low-cost lighting can be used. Power for lighting, such as LEDs, can generally be transmitted via USB or a 1394 bus in any manner.

[0135] The user can also point or signal with an object such as C15 having data C16 thereon such as a retro-reflective dot C16 or a line target C17.

[0136] It is possible to extend the 2D position sensing described above to 3, 4, 5, and 6 dimensions (x, y, +z, pitch, yaw, roll). Two sensing possibilities out of many are described herein in various embodiments. 1. The first possibility, illustrated in Figures 14a and 14b, is to utilize one camera but multiple distinct features or other targets on the object that can provide a multi-degree-of-freedom solution. In one example, the target spacing on the object is known in advance and can be entered into the computer manually or automatically from software containing data about the object, or determined via a taught decision process. 2. The second possibility is the two-camera solution shown in Figures 14c and 14d, which does not require prior knowledge of the target and can actually find the 3D position of a single target by itself, useful for determining the position of a fingertip, for example. For six degrees of freedom of information, even for a line target, at least three points are required, and a combination of lines and points can also be used.

[0137] FIG. 14b illustrates an embodiment of 3-D (three-dimensional) sensing utilizing one stereo camera with three or more data points on the object to be sensed, or in another example on the user's wrist.

[0138] As shown, the user holds an object C30 in his right hand C29, which has at least three visual data C32, C33, and C34, which are viewed by a TV camera C40 whose signals are processed by a computer C41 that also controls a projection display C42. The TV camera C40 also views three other data C45, C46, ​​and C47 on the user's left wrist C48 to determine the pointing direction and approximate orientation of the left hand C51, or its orientation relative to the object C30, or any other data (e.g., data relative to the screen position or other position relative to the TV camera's mounting position), or the user's head if viewed, or anything else. The position and orientation of the object and hand can be determined from three points in the camera image using known photogrammetric equations (see Pinckney, Ref. U.S. Patent No. 4,219,847 and other references in the cited literature).

[0139] Alternatively, for three separate point targets, a colored triangular target can be used, for example, where the intersection points of lines fitted to either side of it define the target data as described below.

[0140] It is also possible to use the camera C40 to view other objects of interest as well. The direction in which the user points at an object C55 represented on the display device C42 is determined, for example, by data C50 on the fingers C52 of the user's left hand C51 (the position and tilt of the wrist can also be determined).

[0141] Alternatively, the fingers can also be detected from just their general gray level image and easily identified relative to the targeted wrist position (especially when the user is clenching his other fingers as shown, so that only finger C52 is extended on the hand).

[0142] The computer can process the gray level image using known techniques, such as the BLOB and other algorithms found on Matrox brand Genesis image processing boards for PCs, and use knowledge of the wrist derived from the data to determine finger pointing direction, allowing the fingers C50 of the left hand to alternatively point at (or touch) a point to be determined on an object C30 also held in the right hand.

[0143] Figure 14c Figure 14c illustrates another version of the embodiment of Figures 14a and 14b, in which two camera "binocular" stereo cameras C60 and C61, processed by a computer C64, are used to image an artificial target (in this case a triangle; see also Figure 2) C65 at the end of a pencil C66, and optionally improve the pointing resolution of a target C67 at the end of the pencil, typically a known small distance from the tip (for clarity, the user and their hand holding the pencil are not shown). This imaging allows tracking of the pencil tip position to determine where on the paper (or TV screen, in the case of a touchscreen) the pencil is touching.

[0144] It may be desirable to have independently controllable, approximately coaxial light sources C62 and C632, controlled by the computer C64 shown, to provide illumination of the retroreflective target independently for each camera. This is because retroreflectors reflect differently at different approach angles, and because the cameras are often angularly spaced (e.g., by a non-zero angle A), they will not see the same target.

[0145] Numerous other camera placements, processing, computational, and other issues are generally discussed in connection with accurately determining the location of an object using a two or more camera stereoscopic system in the above-referenced paper by SFEI Hakim and other references cited therein.

[0146] The computer can also acquire a stereoscopic image of the paper and the targets at the four corners C71 to C74. The solution of the photogrammetry equations allows the position of the paper relative to the camera in space to be determined, and therefore the position of the pencil relative to the paper, in particular the position of its tip, which can be communicated to a display means C75 or other computer program. Even if there is no target at the edge, the pointing direction can be determined by knowing the length of the target C65 and the calculated position of the pencil tip.

[0147] A line target C76 may be useful on the pencil, or multiple circumferentially spaced line targets may also be useful to define the pencil pointing direction from the stereo image pair.

[0148] The working envelope of the measurement system is shown by dotted line C79, which in this case is the area on and above the desktop where the sensor system can operate effectively. Typically, this is sufficient for the task at hand. Note that due to possible camera tilt and other geometric considerations, the effective working envelope for any accuracy or resolution criterion does not necessarily have parallel sides.

[0149] Note that the two-camera (stereo pair) system of Figure 14 has been extensively tested and can provide extremely accurate position and orientation information in up to six degrees of freedom. One particular version, using a commercially available CCD monochrome camera, a Matrox "Genesis" frame grabber board, an image processing board, and suitable stereo photogrammetry software running on an Intel Pentium 300 MHz-based computer, has characteristics that make it well-suited for input from, for example, a large desktop CAD station. It provides 30 Hz updates of six-axis (xyz, roll, pitch, and yaw) data over a range of motion of 0.5 meters by 0.5 meters in x and y (a desktop with the camera pointing directly overhead at the desk) and 0.35 meters in z above the desk, all to an accuracy of 0.1 mm or better, when using, for example, data from a clearly visible round retroreflection (Scotchlite 7615-based) approximately 5-15 mm in diameter on the object. This can be sufficiently accurate for precision tasks such as designing an object in a 3D CAD system.

[0150] The camera in this example is mounted overhead. If it is mounted to the side or front, or at an angle such as 45 degrees to the desktop, the z-axis points outward from the camera.

[0151] Figure 14c further illustrates the stereoscopic arrangement of two cameras used in this case to determine the position and orientation of an object with a line target and data on a portion of a user. Here, cameras C60 and C61 are positioned to view a retro-reflective line target C80, in this case running down a portion of the length of a toy sword blade C81. The line target in this case is fabricated as part of a plastic sword and takes the form of a box-like reflector molded into the corner similar to the reflectors in car taillight reflectors. It can also be fabricated to be a unique color relative to the rest of the sword, and the combination of the two provides an unmistakable indication.

[0152] Typically, when viewed retroreflectively, there are no other bright lines in any typical image. This also illustrates how the shape of a target (e.g., a line) can be used to identify unwanted other glints and reflections that may contain some bright pixels of value in an image. Note that a line-type target may be cylindrical in shape if wrapped around a cylindrical object, so that it can be viewed from multiple angles.

[0153] Aligning the two camera images and solving the photogrammetric equations gives the pointing direction of the line target. If additional points such as C82 are used, a full six-degree-of-freedom solution is available. Also shown here is yet another point, C83, which serves two purposes: it allows for an improved photogrammetric solution, and it serves as a redundant target if C82 cannot be seen due to ambiguity, obliteration, or other reasons.

[0154] This data is calculated in computer C64 and used to modify the display as desired.

[0155] In one embodiment, a Matrox Genesis frame processor card on an IBM 300 MHZ PC was used to read both cameras and process the information at the camera frame rate of 30 Hz. Such line targets are very useful for marking sleeves, pointing glove seams, hat rims, and other decorative and practical purposes, such as outlining the edge of an object or part of it, such as a hole or tear.

[0156] Typically, cameras C60 and C61 have equal magnification and field of view, and the desired measurements overlap. The camera axes may be parallel, but are angled at an acute angle A to one another to increase the overlap of their fields of view for operations at ranges less than 2-3 meters, especially when a large baseline distance d is used to increase accuracy (although the z-range may be smaller). For example, for a CAD drafting application, with a baseline of 0.5-1 meter, A may be 30-45 degrees. For a video game such as that shown in Figure 5, the z-range may be 5 meters or more, and the angle A and baseline will be smaller, allowing for a larger range of operation.

[0157] Database The data on an object, in relation to other aspects of the object and other data, may be known by selling or otherwise providing the user with an object designed with such knowledge and including with it a CD ROM disk or other computer-interfaceable storage medium already having this data. Alternatively, the user or a person may teach this information to a computer system. This is particularly useful when the data is applied by the user to any object.

[0158] Figure 14d Illustrated here is the process used in various embodiments for single point detection to generate commands, where the position (or change in position, i.e., movement) of a fingertip with an attached retro-reflective target is detected by a stereo pair of TV cameras that uses a detection algorithm based in the simplest case on image thresholding, so that only the bright target designation is seen by the finger (and optionally any object associated with it, such as a screen to be touched).

[0159] If this is insufficient to unambiguously define the data on the finger, additional algorithms known in the art can be utilized (many of which are commonly found on image analysis frame grabber boards such as the Matrox Genesis). The processing can include: detecting brightness compared to surroundings or immediate vicinity (contrast); A shape detection step in which a search for shapes such as circles, rings, triangles, etc. is carried out; a color detection step, where a search for a specific color is performed; A movement step in which only target candidates that have moved from their position in the previous TV image are seen.

[0160] Each step may process only what has passed the previous step, or each may be performed independently and the results compared later. The order of these steps may be changed, but modifications are made to each to further identify valid finger target indications.

[0161] The location of the targeted finger is then determined by comparing the location of the finger target in the two camera images of the stereo pair. In this case, there is no alignment problem because a single target is used, which only appears when a point is found in each image.

[0162] After the image of the tip of the finger (or other tool) is found, its position is calculated relative to the screen or paper, and this data is input into the computer controlling the display to change it, for example, the position of a drawn line, an icon, or to determine the vector of movement on the screen.

[0163] Motion detection The computer 8 can be used to analyze the incoming TV image-based signal to determine which points in the image are moving. This is useful for removing stationary background data, since often only moving items, such as hands or objects, are of interest. Furthermore, the direction of movement is often the desired answer, or the fact that movement occurs at all.

[0164] A simple way to determine this is to subtract an image of a high-contrast retro-reflective target from the first image, and then determine which parts are different, essentially representing the movement of the points. Small changes in lighting or other effects will not be registered. Obviously, more sophisticated algorithms exist as well.

[0165] Motion preprocessing is useful when the target contrast is not very high, as it allows a person to remove irrelevant areas and focus all target recognition and measurement processing on the actual target item.

[0166] Such processing is also useful when two-camera stereo is used, since only moving points are considered in image matching, and the problem arises when there are many points in the field.

[0167] Can an object be considered moving? The answer is yes if it is a game or many other activities. However, there can be an issue of the speed of movement. Perhaps in games it is frame-by-frame, i.e. 30Hz for a typical camera. However, in some cases movement can be defined slower, for example 3Hz for a CAD system input using the designer's deliberate motion.

[0168] Once the movement data is identified, its range can then be determined and if the object is then tracked even if it does not move forward from that point, the range measurement provides a good way to fixate the object using more than just two dimensions.

[0169] One can actually use artificial movement of the target if it does not exist naturally. This can be done by vibrating it. If one or more LEDs are used as targets, they can be made to blink, which also shows up in image subtraction (image with LEDs vs. image without LEDs). The same is true for colored targets, which also show up in color image subtraction.

[0170] Image subtraction or other computer processing operations can also be useful in other senses. It is also possible to subtract the background, activating retro-reflected illumination lights where no retro-reflected targets are present, and then subtracting them. One idea is to simply take a picture of a room or other workspace and then capture the targeted object. Subtraction or something like that seems pretty simple. The end result is that any shiny features in the space that are irrelevant, such as shiny doorknobs, glass, etc., are removed from consideration.

[0171] This can also be done with colored targets by color-based image subtraction, which is particularly useful when the desired color is known in advance (via a teaching mode).

[0172] The flow chart shown in Figure 14d illustrates the following steps: A. Acquire stereo pair images; B. Optionally, pre-process the images to determine if motion is present. If it exists, move to the next step, otherwise do not move to the next step or may move (as desired); C. Thresholding the image; D. If the brightness is insufficient, change the brightness collection parameters such as brightness or integration time; E. Identify the target; F. If identification is not possible, add other steps, such as screening the target for color, shape, or size; G. Determine the centroid or other feature of the target point (in this case, the retroreflective point on the finger); H. If necessary, perform a secondary alignment step; I. Compare positions in the stereo pair to determine the range z, and x, y of the target position; J. An auxiliary step for determining the position of a target on the screen when the position of the screen is not known to the computer program, for example, by determining the target on the screen, e.g., by determining what is projected onto the housing or screen; K. Determine the target's position relative to the screen; L. Determining points in the indicated display program; M. Modify the display and program as desired.

[0173] Figure 14e The following is a description of multi-degree-of-freedom image processing of a triangular-shaped color target (disclosed in some embodiments herein), which can be found optically using one or more cameras to obtain the three-dimensional position and orientation of the target using the computer-based method described below. Advantageously, it uses color processing as well as a large number of pixels for the highest resolution, and is typically best for targets defined by a large number of pixels in the image plane because the target is large, or the camera is close to the target, or the camera's field of view consists of a very large number of pixels.

[0174] The method is simple but unique in that it 1) increases accuracy to varying degrees (at the expense of speed), 2) can be applied with more than one camera (more cameras increase accuracy), and 3) can use a combination of target color and triangle(s) to identify the tool or object. It exploits triangular edges to achieve precise sub-pixel accuracy. If the triangle edges can even have gentle curves, the method can still work well. Other geometric shapes can sometimes be handled similarly.

[0175] The method is based on accurately finding the three vertices (F0, G0, F1, G1, F2, G2) of each triangle in the camera field of view by precisely defining the edges and then calculating the intersection points of these edge curves. This is generally more accurate than finding three or four points from the centroid area. However, the choice of which to use often comes down to which is more satisfactory for the consumer or durable and reliable in use.

[0176] In a preferred implementation, one or more cameras are used to capture a target consisting of a brightly colored right-angled triangle on a rectangle of different brightly colored background material. The background color and the triangle color should be two colors that are easily distinguishable from the rest of the image. For purposes of illustration, we will describe the background color as bright orange and the triangle as light blue.

[0177] By using the difference between the color of the background and the color of the triangle, the vertices of this triangle can be found very accurately. If there is more than one triangle on the target, a weighted average of the position and orientation information can be used to improve accuracy.

[0178] Starting from the pixel location of the triangle's centroid from the previous frame, the method begins searching for pixels with the background color or the triangle color. Once a pixel with the triangle's "light blue" color is found, the program proceeds in four opposite directions until each advance finds a color that represents the edge separating the triangle and the "orange" background. The method then extends this edge and defines the three sides of the triangle using a least-squares method. The intersection of the resulting three lines is found, which serves as a rough estimate of the triangle's vertices. These can serve as input for applications that do not require high accuracy.

[0179] If better accuracy is desired, these tentative lines are then used as the starting point for the sub-pixel refinement process. Each of these three lines is checked to see if it is mostly horizontal. If the line is mostly horizontal, a new line is determined by fitting a best fit curve over the pixels in each column that spans the tentative line. If the line is mostly vertical, the same process proceeds over the pixels in the row.

[0180] The color of each pixel the line crosses is translated into a corresponding numeric value. A completely light blue pixel would be given a value of 0, while a completely orange pixel would be given a value of 1. All other colors generate a number between 0 and 1 based on the relative amounts of light blue and orange in it. This numeric value, V, assigned to a pixel is a weighted average of that pixel's color components (R, G, B values, etc.). If the calibrated light blue components are AR, AG, AB, and the orange components are OR, OG, OB, and the pixel components are PR, PG, PB, then the numeric value V is: V=WR*CR+WG*CG+WB*CB where WR, WG, and WB are weighting constants between 0 and 1, and CR is defined as follows: The same process can be used to define the CG and CB.

[0181] This value V is compared to an ideal value U, which is equal to the percentage of orange calculated assuming the angle of the provisional line is the same as that of the ideal line. For example, a pixel that is intersected exactly in the middle by the line will have a U of 0.5 because it is 50% light blue and 50% orange. Fitting the UVs in the columns (or rows) near the provisional line crossings results in a new estimate of the true edge crossing locations. Finally, these sets of intersections can be fitted to lines or gentle curves for each of the three edges, and three vertices can be calculated from the intersections of these lines or curves.

[0182] These three exact vertices can be used in the camera plane (F0, G0, F1, G1, F2, G2) along with the lens formula (herein we will simply use the lens formula for simplicity) to relate the x and y of the target to F and G. F=λX / Z; G=λY / Z λ is the focal length, and z is the vertical distance from the lens to the target location. A triangle on the target is initially defined as lying in a plane parallel to the lens plane. A preferred configuration has a right triangle with right angles defined by x0, y0, and z0, with one side (of length A) extending along the camera's F-axis and the other side (of length B) extending along the camera's G-axis. The orientation of the real target is related to this orientation using Euler angles φ, θ, and ψ. Along with the lens equation and Euler equation, the six derived data values ​​of the three vertices (F0, G0, F1, G1, F2, G2) can be used to define six values ​​of the target's position and orientation. The position and orientation of a point of interest on any tool or object rigidly attached to this target can be easily calculated from the calibration data and conventional translation and rotation transformations. Refinement to deal with lens distortion can be handled by forming a correction function using calibration data that corrects the position of the F and G data. The Euler equations are nonlinear. We linearized them by first assuming that these angles have not changed much since the last video frame. Therefore, we replace φ with φ(old) + U1, θ with θ(old) + U2, ψ with ψ(old) + U3, and z0 with z0(old) + U4, i.e.: φ=φ+U1 θ=θ+U2 ψ=ψ+U3 z0=z0+U4 Substituting these into the Euler equation and applying the lens formula, we obtain the matrix equation SU=R This can be solved for the U value using standard methods such as the Gauss-Jordan routine. The angles and z0 can be iteratively updated until convergence is achieved. The coefficients of this matrix are s11=-A(cos(φ)(F1 / λcos(ψ)+sin(ψ))-sin(φ)cos(θ)(F1 / λsin(ψ)-cos(ψ))) s12=Asin(θ)cos(φ)(F1 / λsin(ψ)-cos(ψ) s13=A(sin(φ)(F1 / λsin(ψ)-cos(ψ))-cos(φ)cos(-θ)(F1 / λcos(ψ)-sin(ψ))) s14=(F0-F1) / λ s21=A(G1 / λ(-cos(φ)*cos(ψ)+sin(φ)sin(ψ)cos(θ))+sin(θ)sin(φ)) s22=Acos(φ)(G1 / λsin(θ)sin(ψ)-cos(θ)) s23=G1 / λA(sin(ψ)sin(φ)-cos(ψ)cos(θ)cos(φ)) s24=(G0-G1) / λ s31=0 s32=-Bcos(θ)(F2 / λsin(ψ)-cos(ψ)) s33=-Bsin(θ)(F2 / λcos(ψ)+sin(ψ)) s34=(F0-F2) / λ s41=0 s42=-B(G2 / λsin(ψ)cos(θ)+sin(θ)) s43=-BG2 / λsin(θ)cos(ψ) s44=(G0-G2) / λ Defined as r1=(F1-F0)z0 / λ+A(F1 / λ(cos(ψ)sin(φ)+cos(θ)cos(φ)sin(ψ))+sin(ψ)sin(ψ)-cos(θ)cos(φ)cos(-ψ)) r2=(G1-G0)z0 / λ+A(G1 / λ(cos(ψ)sin(φ)+cos(θ)cos(φ)sin(ψ))+sin(θ)cos(φ)) r3=(F2-F0)z0 / λ+Bsin(θ)(F2 / λsin(ψ)-cos(ψ)) r4=(G2-G0)z0 / λ+B(G2 / λsin(θ)sin(ψ)-cos(θ)) defined as follows. After the rest of the parameters x0 and y0, the equation: x0=F0z0 / λ Y0=G0z0 / λ is defined as follows:

[0183] A noticeable color transition can provide significantly more information than a black-and-white transition, making it useful for the purpose of accurately calculating an object's position and orientation. As color cameras and high-capacity processors become cheaper, the additional information provided can be accessed at virtually no additional cost. And, very importantly, color transitions are often more pleasant for the user to view than plain black and white. Additionally, the color can vary within a target to create additional opportunities for statistically enhancing the resolution at which the target can be found.

[0184] (Issues in 3D input to computers) Today, computer input for three-dimensional (3D) information is often laboriously performed using a two-dimensional device such as a mouse or similar device. This method is unnatural, both to the person and to the program and its interaction with the person, and CAD designers working with 3D design systems require years of experience to acquire the skills necessary to design effectively using the method.

[0185] A similar situation exists with respect to very common computer video games, which have similar limitations, although their content has become much more three-dimensional and graphically based: these games, too, have not heretofore been natural to the player(s).

[0186] "Virtual reality" also requires 3D input for head tracking, body part movement, etc. This has led to the development of further areas of sensor capability which has resulted in some solutions, but these are either cumbersome for the user, expensive, or both.

[0187] The limitations of 3D computer input have also limited its use in natural-type situations, such as medical teaching, simulations, etc. It further limits young children, the elderly, and people with disabilities from benefiting from computer-assisted living and work.

[0188] Another aspect is digitizing object geometry: sometimes you want to have a plastic model or a real-world part as a starting point for your 3D design.

[0189] We propose one single inexpensive device that can provide all of these controls and can serve as a drawing pad, or input 3D modeled shapes, or even let the user use real clay and have the computer record the new shape as the user sculpts with the real clay.

[0190] Various embodiments link physical activities and body parts to computer commands. A novice user can design a house using a set of purposeful models or "toy" doors, windows, walls, etc. By touching the appropriate toy element and then moving or rotating the user's hand, the user can place the element in the appropriate position. The user can get their visual cue by either seeing the toy's position on a table or seeing a corresponding scale representation on a computer display. Many other embodiments are possible.

[0191] (Object Tracking) In one general aspect, a method for tracking an object of interest is disclosed. The method includes acquiring a first image and a second image representing different viewpoints of the object of interest, processing the first image into a first image data set and processing the second image into a second image data set. The method further includes processing the first image data set and the second image data set to generate a background data set related to a background, and generating a first difference map by measuring a difference between the first image data set and the background data set and a second difference map by measuring a difference between the second image data set and the background data set. The method also includes detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map, and generating an absolute position of the object of interest from the first and second relative positions of the object of interest.

[0192] Processing the first image into a first image data set and the second image into a second image data set may include determining an active image region for each of the first and second images, and extracting an active image data set from the first and second images contained within the active image region. Extracting the active image data set may include one or more of cropping the first and second images, rotating the first and second images, or shearing the first and second images.

[0193] In one implementation, extracting the valid image data set may include arranging the valid image data set into an image pixel sequence having rows and columns, and the extracting step may further include identifying a maximum pixel value within each column of the image pixel sequence and generating a data set having one row, with the identified maximum pixel value for each column representing that column.

[0194] Processing the first image into a first image data set and the second image into a second image data set may also include filtering the first and second images. The filtering may include extracting edges in the first and second images. The filtering may further include processing the first and second image data sets to enhance differences between the first image data set and a background data set, and to enhance differences between the second image data set and a background data set.

[0195] Processing the first image data and the second image data to generate the background dataset may include generating a first set of one or more background datasets associated with the first image dataset, and generating a second set of one or more background datasets associated with the second image dataset.

[0196] Generating a first set of one or more background data sets may include generating a first background set representing a maximum value of data in a first image data set representing the background, and generating a second set of one or more background data sets includes generating a second background set representing a maximum value of data in a second image data set representing the background. The generating may further include increasing values ​​included in the first and second background sets by a predetermined value for the first and second background sets representing a maximum value of data representing the background.

[0197] Generating a first set of one or more background data sets may include generating a first background set representing a maximum value of data in the first image data set representing the background, and generating a second set of one or more background data sets may include generating a second background set representing a minimum value of data in the second image data set representing the background. The generating may further include lowering values ​​included in the first and second background sets by a predetermined value for the first and second background sets representing the minimum value of data representing the background.

[0198] Generating the first set of background data sets may include sampling the first image data set, and generating the second set of background data sets may include sampling the second image data set, wherein the sampling may occur automatically at predetermined time intervals, and wherein each sample may include data not associated with the background.

[0199] Generating a first set of one or more background data sets may include maintaining a plurality of samples of the first image data set within each background data set, and generating a second set of one or more background data sets may include maintaining a plurality of samples of the second image data set within each background data set.

[0200] Generating each first background data set may include selecting a value from the plurality of samples that represents a background for each element in the first image data set, and generating each second background data set may include selecting a value from the plurality of samples that represents a background for each element in the second image data set, and the selecting may include selecting a median value from all sample values ​​in each of the background data sets.

[0201] In other implementations, the generating may include comparing the first image dataset to a subset of the background dataset, and comparing the second image dataset to a subset of the background dataset.

[0202] In other implementations, generating the first difference map may further include representing each element in the first image dataset as one of two states, and generating the second difference map may further include representing each element in the second image dataset as one of two states, where the two states represent whether the value is consistent with the background or not.

[0203] In yet another implementation, the detecting may include identifying a cluster in each of the first and second difference maps, where each cluster has an element, the state of the element in its associated difference map indicating that the element is inconsistent with the background.

[0204] Identifying a cluster may further include reducing the difference map to one row by counting elements in the column that are inconsistent with the background. Identifying a cluster may further include identifying a column as being in the cluster and classifying adjacent columns as being in the cluster. Identifying a column as being in the cluster may also include identifying a central column.

[0205] Identifying the clusters may further include identifying locations associated with the clusters. Identifying locations associated with the clusters may include calculating a weighted average of the elements in the clusters.

[0206] Detecting may further include classifying the cluster as the subject of interest. Classifying the cluster may further include counting the elements in the cluster and classifying the cluster as only the subject of interest if the count exceeds a predetermined threshold. Classifying the cluster may further include counting the elements in the cluster and counting the number of total elements classified as inconsistent within the background in the difference map, and classifying the cluster as only the subject of interest if the ratio of the count of elements in the cluster to the number of total elements exceeds a predetermined threshold.

[0207] The detecting step may further include identifying a sub-cluster within the cluster that represents a pointing end of the object, and identifying a portion of the sub-cluster.

[0208] In the above implementations, the object may be a user's hand, and the method may include controlling an application program using an absolute portion of the object.

[0209] The above implementation may further include acquiring a third image and a fourth image representing different perspectives of the object, processing the third image into a third image data set and the fourth image into a fourth image data set, and processing the third image data set and the fourth image data set to generate a background data set related to the background. The method may also include generating a third disparity map by determining a difference between the third image data set and the background data set, and generating a fourth disparity map by determining a difference between the fourth image data set and the background data set, and detecting a third relative position of the object in the third disparity map and a fourth relative position of the object in the fourth disparity map. An absolute position of the object may be generated from the first, second, third, and fourth relative positions of the object.

[0210] As part of this implementation, the object may be the user's hand, or may involve controlling an application program using the absolute position of the object.

[0211] In another aspect, a method for tracking an object of interest controlled by a user in conjunction with a computer is disclosed. The method includes acquiring images from at least two viewpoints, processing the acquired images to generate an image dataset for each acquired image, and comparing each image dataset with one or more background datasets to generate a difference map for each acquired image. The method also includes detecting a relative position of the object of interest in each difference map, generating an absolute position of the object from the relative positions of the object, and using the absolute position to enable a user to interact with a computing application.

[0212] Additionally, the method may include mapping the absolute position of the object to screen coordinates associated with a computer application and using the mapped position to couple with the computer application. The method may also include recognizing a gesture associated with the object by analyzing changes in the absolute position of the object and combining the absolute position and the gesture to couple with the computer application.

[0213] In another aspect, a multi-camera tracking system is disclosed that interacts with an application program running on a computer. The multi-camera tracking system includes two or more video cameras configured to provide different perspectives of an area and is operable to generate a series of video images. A processor is operable to receive the series of video images and detect objects appearing in the area. The processor performs the following operations: generate a background data set from the video images, generate an image data set for each received video image, compare each image data set to the background data set to generate a difference map for each image data set, detect a relative position of the object in each difference map, generate an absolute position of the object from the relative position of the object, and map the absolute position to a position indicator associated with the application program.

[0214] In the above implementation, the object may be a human hand. Further, the region may be defined in front of a video display associated with a computer. The processor is operable to map the absolute position of the object to a position indicator on the video display such that the position indicator is aligned with the object.

[0215] The region may be defined at any distance in front of a video display associated with a computer, and the processor may be operable to map the absolute position of the object to a position indicator on the video display such that the position indicator is aligned with a position pointed to by the object. Alternatively, the region may be defined at any distance in front of a video display associated with a computer, and the processor may be operable to map the absolute position of the object to a position indicator on the video display such that movement of the object scales with larger movements of the position of the position indicator on the video display.

[0216] The processor may be configured to emulate the functionality of a computer mouse, which may include configuring the processor to emulate controlling the buttons of a computer mouse using gestures derived from movements of the object. Sustained positioning of the object for a predetermined period of time may trigger a selection action within an application program.

[0217] The processor may be configured to emulate controlling the buttons of a computer mouse based on a maintained position of the object for a predetermined period of time. Maintaining the position of the object within the interactive display area for a predetermined period of time may trigger a selection action within an application program.

[0218] The processor may be configured to emulate controlling the buttons of a computer mouse based on a sustained position of the position indicator within the interactive display area for a predetermined period of time.

[0219] In the above-described aspects, the background data set may include data points representing at least a portion of a static structure. In this implementation, at least a portion of the static structure may include a patterned surface that is visible to the video camera. The static structure may be a window frame. Alternatively, the static structure may include a strip of light.

[0220] In another aspect, a multi-camera tracking system for interacting with an application program running on a computer is disclosed. The system includes two video cameras configured to provide different perspectives of an area and is operable to generate a sequence of video images. A processor is operable to receive the sequence of video images and detect objects appearing in the area. The processor generates a background data set from the video images, generates an image data set for each received video image, compares each image data set to the background data set to generate a difference map for each image data set, detects a relative position of the object in each difference map, generates an absolute position of the object from the relative position of the object, identifies a sub-area indicated by the object, and, if the object occupies the identified sub-area, executes processing to associate an action with the identified sub-area being activated and apply the action to interact with the application program.

[0221] In the above implementation, the object may be a human hand. Further, the action associated with the identified sub-region may emulate the activation of a keyboard key associated with an application program. In a related implementation, maintaining the position of the object in any sub-region for a predetermined period of time may trigger the action.

[0222] The details of one or more implementations are set forth in the accompanying drawings and the description below.

[0223] 15 shows a multi-camera motion tracking and control system D100 interacting with an image viewing system. In this implementation, two cameras D101 and D102 scan an area of ​​interest D103. A controlled or known background D104 surrounds the area of ​​interest D103. The area of ​​interest D105 is tracked by the system if it enters the area of ​​interest D103. The object of interest D105 is any general object inserted into the area of ​​interest D103, typically the hand or finger of a system user. The object of interest D105 may also be a selection device such as a pointer.

[0224] The sequence of video images acquired from cameras D101 and D102 is conveyed to a computing device or image processor D106. In this implementation, the computing device is a general-purpose computer running additional software that provides feedback to the user on a video display D107.

[0225] FIG. 16A shows a typical implementation of a multi-camera control system D100. Two cameras D101 and D102 are located outside the region of interest D103. The cameras are oriented so that the intersection D204 of their fields of view (D205 for camera D101, D206 for camera D102) encompasses the region of interest D103. The orientation is such that the cameras D101 and D102 are rotated on axes that are approximately parallel. In this example, the floor or window ledge and sidewall provide a controlled background D104 with prominent edges. The corresponding field of view captured by camera D101 is shown in FIG. 16B. Although not shown, the field of view captured by camera D102 is a mirror image of the field of view captured by camera D101. The controlled background D104 does not have to cover the entire field of view D205 of the cameras. For each camera, the active image region D208 is found to be entirely contained within the controlled background D104 and to encompass the entire region D103. The background D104 can be modeled such that the object D105 is controlled to differ in its characteristics from the background D104, either partially or completely. When the object D105 appears within the region D103, it occludes a portion of the controlled background D104 within the active image region D208 of each camera D101, D102. At that point of occlusion, either fully or partially, the occluded image is inconsistent with the model of the controlled background D104 with respect to selected characteristics.

[0226] In summary, the object of interest D105 is identified and, if found, its position within the active image region D208 of both cameras is calculated. Using the position data of each camera D101, D102, and the positions of the cameras relative to the region of interest D103, as well as parameters describing the cameras, the position of the object of interest D105 within the region of interest D103 is calculated.

[0227] The processing performed by the image processor D106 (FIG. 15), which may be implemented via software processing or hardware, is shown diagrammatically in FIG. 17. Camera images are simultaneously carried from cameras D101, D102 and captured by image acquisition modules D304, D305 (respectively) and transferred to image buffers D306, D307 (respectively) within the image processor D106. Image detection modules D308, D309 independently detect the object of interest D105 in each image and determine its position relative to the camera field of view. Relative position information D310, D311 from both camera fields of view is combined by a combination module D312 and, if necessary, refined by a position refinement module D313 to determine the global presence and position of the object of interest D105 within the region of interest D103 in block D314. If desired, specific gestures performed by the user may be detected in a gesture detection module D315. The results of the gesture detection process are then conveyed to another process or application D316, either on the same image processor D106 or on another processing device. The process of gesture detection is described in further detail below.

[0228] The image detection modules D308 and D309 are identical in the processing they perform. An implementation of these image detection modules D308, D309 is shown in FIG. 18. In block D402, the image processor D106 extracts image data corresponding to the active image region D208 (of FIG. 16B) from the captured image data stored in the image buffer D306 or D307. The image may be filtered in a filtering process D403 to highlight or extract image aspects or features where the background D104 and the object of interest D105 are different but are otherwise consistent within the background D104 over time. In some implementations, data representing the active image region may also be reduced by a scaling module D104 to reduce the amount of computation required in subsequent processing steps. Using the resulting data, the background D104 is modeled by one or more instances of a background model process in block D405 to generate one or more descriptions, represented as background model data 406, of the controlled background D104. Therefore, the background D104 is modeled with respect to the desired appearance or characteristics of that image. The background model D406 is converted into a set of criteria in process D407. In a comparison process D408, the filtered (from process D403) and / or reduced (from module D404) image data is compared to these criteria (from process D407), and locations where the current data is inconsistent with the background model data D406, i.e., locations where the criteria are not met, are saved in an image or difference map D409. In detection module D410, the difference map D409 is analyzed to determine whether any such inconsistency qualifies as a possible indication of the object D105 of interest and whether these criteria are met, and its location within the camera field of view (D205 or D206) is determined.The position of the object 105 may be further refined (if necessary) in block D411 to generate a presence relative to the camera and a position output D310 or D311 relative to the object D105 (as described above in connection with Figure 17).

[0229] In block D402 of FIG. 18, the image processor D106 extracts image data corresponding to the active image region D208 (of FIG. 16B). Image data may be extracted by cropping, shearing, rotating, or transforming the captured image data. Cropping extracts only the portion of the entire image that is within the active image region D208. A bound is defined, and any pixels within the bounds are copied, unmodified, to a new buffer, while pixels outside the bounds are ignored. The active image region D208 may be of any shape. Shearing and rotating rearranges the data into a more convenient order for further processing, such as a rectangle, so that it can be addressed in terms of rows and columns of pixels.

[0230] Rotation changes the contents of an image so that it appears as if the image is rotated. Rotation rearranges the positions from (x,y) to (x',y') according to the following equation: × (times) × θ × θ × θ × θ × θ × θ FUNCTION EQU00001, where θ is the angle by which the image is rotated.

[0231] When cameras D101 and D102 are properly mounted relative to the region D103, the desired angle of rotation is usually small. When the desired angle of rotation is small, shear may be used to provide a computationally simpler approximation than rotation. Shear distorts the shape of the image, so that the deformed shape appears as rows and columns slid above and below each other. Shear rearranges the pixel positions according to the following equation: ".Function..x. .x..x. ".Function..x. ##EQU00002##, where shx represents the amount of horizontal shear in the image and shy represents the amount of vertical shear in the image.

[0232] Implementation of the multi-camera control system D100 applies to situations where the object D105, either in whole or in part, has either a higher or lower brightness than the controlled background D104. For example, the D104 background may be illuminated to create this condition. The filtering block D403 passes brightness information associated with the image data. A single background model D406 represents the expected brightness of this background D104. In practice, the brightness of the controlled background D104 may vary within the active image area D208, and therefore the background model D406 may store expected brightness values ​​for all pixels within this active image area D208. The comparison reference generation process D407 corrects each brightness value from the background model D406 to account for signal noise (beyond what can be calculated in the background model) and slight variations in brightness of the controlled background D104, thereby generating the minimum brightness value that can be classified as consistent with the background model D406. For example, if the brightness of the controlled background D104 is higher than the brightness of the object of interest D105, then processing block D407 reduces the brightness value of each pixel by an amount greater than the expected magnitude of signal noise and brightness variation.

[0233] In some implementations of the system D100, the region of interest D103 is sufficiently narrow that it may be modeled as a planar region. The direction of that plane is parallel to the front and back faces of the dotted cube representing the region of interest D103 in FIG. 15. If two conditions are met: (1) the object of interest D105, when detected, occludes the background D104 in all rows and some columns of the active image region D208; and (2) a single set of values ​​in the background model D406 fully characterizes an entire column of pixels in the active image region D208, the active image region D208 may be reduced to a single row of pixels in the optional scaling model D404. The first condition is typically met when the active image region D208 is thinner than the object of interest D105. The second condition is met by implementation of blocks D403, D405, D406, and D407 described above. The application of the scaling module D404 reduces the complexity of the processing that needs to be performed later in the processing, as well as reducing the storage requirements of the background model D406.

[0234] The particular implementation of the scaling module D404 depends on the specifications of processing blocks D403, D405, D406, and D407. If the luminance of the controlled background D104 is expected to be higher than the luminance of the object of interest D105, as described above, one implementation of the scaling module D404 represents each column by the highest luminance in that column. That is, for each column, the highest value in that column is copied to a new array. This process has the added advantage that the high luminance portions of the controlled background D104 do not have to fill the entire controlled background D104.

[0235] An alternative implementation applies to situations where the controlled background D104 is static, i.e., does not contain motion, but is not brightness-limited. A sample source image is included as an example in FIG. 19. In this case, the object of interest, as sensed by the camera, may contain brightness values ​​that are also found in the controlled background D104, or may be close to them. In practice, brightness variations in the controlled background D104 (e.g., caused by a user moving in front of the device and thereby blocking some ambient light) may be significant in magnitude relative to the difference between the controlled background D104 and the object of interest D105. Therefore, a specific type of filter may be applied in the filtering process D403 to produce a consistent result or a result that de-emphasizes variations in overall brightness, while emphasizing parts of the object of interest D105. A 3×3 Prewitt filter is typically used in the filtering process D403. FIG. 19B shows the result of a 3×3 Prewitt filter on the image on FIG. 19A. In this implementation, two background models D406 may be maintained, one representing each of the high and low values, and both representing the range of values ​​expected for the filtered pixels. A comparison criteria generation process D407 then reduces the low values ​​and raises the high values ​​by an amount greater than the expected magnitude of signal noise and brightness variation. The result is a set of criteria, an example of which for low values ​​is shown in FIG. 19C and for high values ​​in FIG. 19D. These corrected images are passed through a comparison process D408, which classifies pixels as inconsistent with the controlled background D104 if their values ​​are either lower than the low value criteria (FIG. 19C) or higher than the high value criteria (FIG. 19D). The result is a binary difference map D409, an example of which, corresponding to FIG. 19B, is shown in FIG. 19E.

[0236] The preceding implementations allow for the use of, for example, many existing surfaces, such as walls, window frames, etc., as the controlled background D104, where those surfaces may have any brightness, texture, edges, or lines of light fixed to the surface of the controlled background D104. The above implementations also allow for the use of a controlled background D104 that includes, for example, a predetermined pattern or texture, where the above process detects the absence of the pattern in areas where the object D105 of interest occludes the controlled background D104.

[0237] The difference map D409 stores the locations of all pixels found by the above-described method to be inconsistent with the background D104. In this implementation, the difference map D409 may be represented as a binary image, where each pixel may be in one of two states. Those pixels that are inconsistent with the background D104 are identified, or "tagged," by setting the pixels in the corresponding row and column of the difference map to one of those states. Alternatively, the corresponding pixels are set to the other state.

[0238] An implementation of the detection module D410 for detecting the object of interest D105 in the difference map D409 is shown in FIG. 20. Another scaling module in block D603 provides an additional opportunity to reduce the data to a single-dimensional array of data and may be applied as needed to situations where the orientation of the object of interest D105 does not have a significant effect on the overall extent of the object of interest D105 within the difference map D409. In practice, this applies to many situations where the number of rows is less than or similar to the number of normal columns occupied by the object of interest D105. When applied, the scaling module in block D603 reduces the difference map D409 to a map of one row, i.e., a single-dimensional array of values. In this implementation, the scaling module D603 may count the number of tagged pixels in each column of the difference map D409. As an example, the difference map D409 of FIG. 21A is reduced in this manner and is shown as graph D709 in FIG. 21B. Applying this optional processing step reduces processing requirements and simplifies some of the subsequent calculations.

[0239] Continuing with this implementation of this detection module D410, it is observed that the tagged pixels in the difference map (D409 in the example of FIG. 31A) associated with the object of interest D105 roughly form a cluster D701, but the cluster is not necessarily connected. A cluster identification process D604 classifies pixels (or columns, if the scaling module D603 has been applied) as to whether they are members of the cluster D701. Various methods for finding clusters of samples exist and may be applied, with the subsequent method being selected based on processing simplicity. Note that if the object of interest D105 is present, the count of correctly tagged pixels will be greater than the number of false positives. Therefore, the median location is expected to be somewhere within the object of interest D105. Part of this implementation of the cluster identification process D604, when applied to a map of one column (e.g., when a scaling module is provided in block D603 or D404), calculates the center column D702 and tag columns as part of the cluster D701 if they are within a predetermined distance D703 corresponding to the largest number of columns they are expected to occupy. Part of this implementation of the cluster identification process D604, when applied to a map of multiple rows, adds tagged pixels to the cluster D703 if they meet the proximity distance criteria.

[0240] In this implementation, a set of criteria is received by the cluster classification process D605 and then applied to the cluster D701 to verify that the cluster qualifies as a match for what is expected for the object D105 of interest. Thus, the process D605 determines whether the cluster D701 should be classified as belonging to the object D105 of interest. Part of this implementation of the cluster classification process D605 calculates counts of tagged pixels within the cluster D701 and calculates a count of all tags. The counts within the cluster D701 are compared to a threshold to eliminate false matches in clusters with few tagged pixels that are expected for the object D105 of interest. Additionally, the ratio of the count of pixels within the cluster D701 relative to the total count is compared to a threshold to further reduce false matches.

[0241] If the cluster D701 passes these criteria, the cluster description is refined in process block D606 by calculating the center of gravity associated with the cluster D701 in process D607. The center location found by scaling module D603 will be within the range defining the object of interest D105, but it is not necessarily the center of the object. The weighted mean D710, or center of gravity, provides a better measure of cluster location and is calculated as needed within process D606 as a subprocess D607. The weighted mean D710 is calculated by the following equation: × Function × Function ##EQU00003##, where {overscore(x)} is the mean, c is the number of columns, and C[x] is the count of tagged pixels in column x.

[0242] The range of the cluster D704 may also be calculated within process D606, if desired, and is shown as process D608. Cluster D703 may contain some false positive outliers, and therefore, as part of this implementation, the range may be defined as including a predetermined percentage of tagged pixels, or, in situations where relatively few pixels are expected to be tagged, including those tagged pixels (or columns, if scaling module D603 is applied) that form a tight sub-cluster, i.e., those tagged pixels (or columns) that have untagged neighbors.

[0243] In addition to the middle and boundary coordinates, the orientation of the object of interest D105 may optionally be inferred by computing the cluster moments, which is represented by the compute cluster orientation process in subprocess D609 within process D606.

[0244] In some applications of the system D100, the object D105 is used as a pointer. In this case, the "pointing edge" of the object D105 is desired and may be determined by the pointing edge calculation subprocess within process D606 if the region D103 contains a sufficient number of rows and if that number of rows has not been reduced. An example is shown in FIG. 21C. The object D105 normally enters or is restricted from entering the active image region D208 from a known boundary of that region. The pointing edge D705 of the object D105 (e.g., the user's fingertip) will be part of the cluster D701 that is furthest from the input region D706 to the active image region D208. The cluster D701 may contain some false positive outliers. Thus, the pointing edge D705 may be defined as the region D707 within the cluster D701 that includes the tagged pixels near the farthest boundary side of the cluster D701, or, in situations where relatively few pixels are expected to be tagged, the farthest tagged pixels that form an adjacent subcluster, i.e., those tagged pixels that have tagged neighbors. This subcluster is identified by the subcluster pointing edge process D610, and the location of the subcluster is found in process D611.

[0245] Continuing with this implementation, the process performed by the smoothing module D612 may be applied as needed to any or all of the locations found in process D606. Smoothing is a process that combines previously solved results with those that move in a stable manner from frame to frame. The weighted average coordinate D710 found by the center of gravity determination process D607 depends on many samples and is therefore inherently stable. The extent D704 found by the cluster boundary area determination process D608 and the pointing edge D705 found by D611 coordinates depend on a relatively small number of clusters, and the state of a single pixel may have a significant impact. Because the size of the area represented by the object of interest 105 is expected to remain relatively stable, smoothing may be applied to the distance between the measured extent D704 in relation to the weighted average coordinate D710 of the cluster. Because the shape and orientation of the object D105 is expected to change more slowly than the overall position of the object D105, smoothing may be applied to the distance of the pointing end D705 measured relative to the weighted average coordinates D710 of the cluster.

[0246] The process used in the center of gravity process is Equation 1: s(t) = (a × r(t)) + ((1-a × s(t-1)). In Equation 1, the smoothed value at time (s(t)) is equal to 1 - the scale value (a) × the smoothed value at time-1 (t-1). This amount is added to the raw value at time t (r(t)) multiplied by a scalar (a) which is between 0 and 1.

[0247] Referring to Figure 22, an implementation of the system D100, as described above, utilizes one or more background models D406 (Figure 22). An implementation of a background model processor or element D405 that generates the background model data D406 is shown in Figure 22. This implementation of the background model element D405 automatically and dynamically generates background models, allowing for unattended operation of the system.

[0248] Input data D802 is provided by the output of the scaling model 404 for this implementation of the background model element D405. Input is available every frame and is sampled in a sampling process D803. This sample may include the object of interest D105, occluding a portion of the controlled background D104. For each pixel, a range of values ​​may better represent the background D104 than a single value. By including the effect of this range in the background model, the expansion in process D407 may be made tighter. Contributing multiple frames of data to the sample makes this range observable, but also increases the portion of the background D104 occluded by the object of interest D105 if the frames are sampled while the object of interest D105 is moving. The optimal number of frames to use depends on the expected movement of the object of interest D105 in the particular application of the system. In practice, for a hand-tracking system, 10 frames, representing approximately 0.33 seconds, is sufficient to observe the majority of the range without allowing the object of interest to move to occlude unnecessary portions of the background. If a particular background model is compared in the comparison process D408 as an upper range on values ​​assumed to match the background D104, then the maximum value of each pixel observed in multiple frames may be recorded as the sample value. If a particular background model D406 is compared in the process D408 as a lower range on values ​​assumed to match the background D104, then the minimum value of each pixel observed in multiple frames may be recorded as the sample value.

[0249] In this implementation of the background model element D405, samples from the sampling process D803 are added to a buffer D804 having storage locations for n samples, where the oldest sample in the history is replaced. This history therefore contains n sampled values ​​for each pixel. Since the span of time d represented in the buffer is rate dependent, new samples are obtained and added to the history r according to Equation 2, written as ##EQU00004## below.

[0250] In this implementation, the central processing block D805 selects for each pixel a value it determines that is characteristic of the controlled background D104 at the location represented by that pixel. One way to select a value characteristic of the controlled background D104 in the processing block D805 is to select the median value of each pixel. For any pixel, the n sampled values ​​in the buffer D804 may represent that object D105. The period d is selected so that that object D105 does not occlude any one pixel of the controlled background D104 for an accumulated period of d / 2 or longer within any time span of d. Therefore, for any pixel, the majority of the samples are characteristic of the background D104, and therefore the median of the sampled values ​​is the value characteristic of the background D104.

[0251] The background model element D405 is adaptive; any changes to the background D104 are reflected in the output of the central processing block D805 once they have been observed for d / 2 times. The system does not require the entire control background D104 to be visible when started; the subject D105 may be shown at start-up, but must be observed for d times before a sample provides output. Optionally, a restriction may be applied such that the subject D105 must be absent when the system is started; in this case, the first observed sample value may be copied to all n samples in the buffer D804, allowing the system to generate output sooner.

[0252] The duration for which any one pixel of the controlled background D104 is occluded by the object of interest D105, and therefore the duration d, depends on the particular application of the system. The number of samples n can be estimated relative to the memory buffer and available processing power.

[0253] The preceding discussion represents one implementation of obtaining the position of the object of interest D105 within and relative to the images acquired by cameras D101 and D102. Once the object of interest D105 is successfully detected and its coordinates found in both camera fields of view D205 and D206 by detection modules D308 and D309 of Figure 17, then a combination of these coordinates is sufficient to recover the position of the object of interest D105 within the range of the object of interest D103. In the implementation described in Figure 17, the position of the object of interest D105 is calculated in combination module D312.

[0254] 23A and 23B, an implementation of the combination module D312 is shown. For each camera D101 and D102, the position D902 of the object of interest D105 p on the camera image plane D904 is converted to an angle D905, referred to in this description as β (.β), measured on a reference plane whose normal is defined by the axis of rotation of the camera D101, D102. (In reality, the axes are not exactly parallel and do not exactly lie in a single plane; however, the process described herein allows for this error.) By approximating the cameras D101, D102 as ideal pinhole models of the cameras, their angles (.β) relative to the vector D906 defining the camera's orientation are estimated.

[0255] As shown in Figure 23A, Equation 3 shows the approximate calculation: β × ##EQU00005##. To approximate the angle β (β), the inverse tangent is applied to the quantity of the focal length (f) divided by the position p on the image plane projected onto the intersection of the reference plane and the image plane.

[0256] For maximum accuracy, the intrinsic camera parameters (position of the principal point and image scale) and the radial distortion caused by the lens should be corrected by transforming the distorted position (as represented by the relative position information D310, D311) to an ideal position. More specifically, the ideal position is the position on the image plane D904 where the object D105 would be projected if the cameras D101, D102 had the characteristics of ideal pinhole cameras (where Equation 3 produces the correct angle). A set of correction equations is presented in Z. Zhang, A Flexible New Technique for Camera Calibration, Microsoft Research, http: / / research.microsoft.com / .about.zhang, which is incorporated by reference. It is understood that for many applications of this system, the approximation described above provides sufficient accuracy without this correction.

[0257] Continuing with the description of the combination module D312, as shown in Figure 23B, a reference vector D907 is defined such that it passes through the positions of both cameras D101 and D102 on a reference plane, where the reference plane is defined such that the axis of rotation of the camera defines the normal to the reference plane. The angle D908 through which the camera is rotated is measured relative to the reference vector D907.

[0258] The formula for measuring an angle is shown in Equation 4: α = β0 + β. The measure of angle α (α) is equal to angle β (β0) and angle β (β).

[0259] Equation 4 is applied to measure the angle D909 of the object of interest D105 relative to the reference vector D907. This angle is referred to herein by the α (.α.) symbol. The angle α D909 relative to each camera D101 and D102, and the length of the reference vector D907, are sufficient to find the position of the object of interest D105 on the reference plane via Equation 5 and Equation 6.

[0260] Equation 5 is: .×. .×..×. .×..α..×..×. .×..α. .×..×. .×..α. .×..×. .×..α. ##EQU00006##The offset (y) is equal to the inverse tangent of the angle (.αA) relative to camera A 101, the tangent of the angle (.αB) relative to camera B D102 multiplied by the vector length D907 (w), the tangent of the angle (.αA) relative to camera A D101, and the tangent of the angle (.αB) relative to camera B D102.

[0261] Equation 6 calculates the offset of the object of interest (xA) as follows: .×. .×..α.##EQU00007##. In Equation 6, the offset (xA) is measured by the offset from Equation 5 (y) divided by the tangent of the angle (.αA) subtended by Camera A D101.

[0262] The position of the object D105 on the axis perpendicular to the reference plane may be found by equation 7 (.×. .×.##EQU00008##), which is applied to the position in each image using the distance of the object D105 from the camera.

[0263] In Equation 7, the position (z) is calculated as the position (p) on the image plane projected onto a vector in the image plane orthogonal to its use in Equation 3 divided by the focal length (f) multiplied by the distance of the object D105 of interest from the camera (l).

[0264] These relationships provide the coordinates of the object of interest D105 relative to camera A D101. Knowing the position and size of the region of interest D103 relative to camera A D101, the coordinates may be transformed so that they relate to the region of interest D103, D312 in Figure 17.

[0265] Smoothing may be applied to these coordinates as needed in the fine-tuning module D313 of this system implementation shown in FIG. 17. Smoothing is the process of combining the results with previously solved results so that motion is stable from frame to frame. One method of smoothing for these specific coordinate values ​​(xA, y, z found by the combination module D312) is described in this invention. The components of the coordinate values ​​associated with the object of interest D105, i.e., x, y, and z, are independently and dynamically smoothed. The degree of dampening S is calculated by Equation 8, where S is dynamically and automatically adjusted in response to changes in position and is calculated as follows: × × ltoreq(≦)α × × α × × × × α × × << × × gtoreq(≧) × × Function Function ##EQU00009##. In Equation 8, s(t) is the smoothed value at time t, r(t) is the raw value at time t, DA and DB are thresholds, and SA and SB define the degree of dampening.

[0266] As shown in Figure 24, two distance thresholds, DA and DB, define three ranges of motion. Position changes, or motion, that are less than DA are significantly damped by SA (D1001), thereby reducing the tendency of values ​​to switch back and forth between two adjacent values ​​(a side effect of discrete sampling of the image). Position changes greater than DB are slightly damped by SB (D1002), or not damped at all. This reduces or eliminates the lag and vagueness introduced in some other smoothing procedures. The degree of damping varies for motion between DA and DB, the region shown as D1003, so that the transition between slight and significant damping is less noticeable. The scalar a applied to Equation 1 is found by Equation 9 as follows: .×. ##EQU00010##In Equation 9, the scalar (a) is restricted to be greater than or equal to 0 and less than or equal to 0, and the dampening value of S is found by Equation 8, where e is the elapsed time since the previous frame.

[0267] These coordinates D314 of the object D105, when found, are typically passed to another process, such as a user application program D316 for use. When performed, they may be passed to another process running on the same image processor D106, as in the calculations described above, or to another computing device. The manner in which the data is passed to the application program D316 may include emulation of conventional user input devices (including a mouse and keyboard), allowing the system to provide existing control functionality within the application program D316. The coordinates D314 of the object D105 may be calculated for every video frame captured by the camera, where a video frame is typically captured 30 or more times per second. This results in little delay between the user's actions and the application's response.

[0268] In a typical implementation of the system, the application program D316 provides feedback to the user by displaying a visual representation of an indicator on the video display D107. The indicator is moved so that its position and movement mimics the movement of the object of interest D105.

[0269] In one variation of this form of user interface, an indicator such as a mouse pointer is displayed in front of other graphics, and its movement is mapped to the two-dimensional space defined by the screen surface. This form of control is similar to that provided by a computer mouse, such as one used with the Microsoft® Windows® operating system. An exemplary feedback image of an application using this type of control is shown at D1102 in Figure 25A.

[0270] 25A (and briefly to FIG. 17), the image processor D106 also includes an optional coordinate remapping process D317 (FIG. 17) that is operable to remap global positioning presence and location coordinates D314 (associated with the object of interest D105) to a location overlaid on the image D1102 with the equivalent of Equation 10 for the x coordinate and an equation for the y coordinate, as follows: <.ltoreq(≦)..ltoreq(≦).> ##EQU00011##.

[0271] In Equation 10, xh is the coordinate location D314 associated with object D105, xc is the on-screen cursor location mapped to 0-1, and bl and br are the locations of the left and right extents of the sub-region within region D103. As shown in FIG. 25B, the entire region of display D1102 is represented by a sub-region D1103 that is entirely contained within region D103. Locations within sub-region D1103 (e.g., A D1105) are linearly mapped to locations within display D1102 (e.g., D1106). Locations outside sub-region D1103 but still within region D103 (e.g., location B D1107) are mapped to the closest location on the boundary of display region D1102 (e.g., D1108). This reduces the likelihood that a user will unintentionally remove the object of interest D105 from the subregion (typically with the user's hand or pointing finger) while attempting to move the indicator D1101 near the border of the display.

[0272] In situations where the region of interest D103 is immediately in front of the video display D107, the sub-region D1103 may be defined to be aligned with the video display D107, so that the indicator D1101 appears aligned with the object of interest D105. If the region of interest D103 is relatively thin, e.g., less than 5 cm, and the sub-region D1103 is defined in this manner, then the system approximates user interaction, i.e., a "touch screen," without being limited by the size of the video display D107 and without requiring direct contact between the user and the video display D107 surface (e.g., the video display and the user may be on either side of a window). As will be appreciated, the system D100 can be used with a variety of video display sizes, including not only computer monitors (CRT or LCD displays), but also rear-projection television monitors, large flat-screen LCD monitors, and forward-projection presentation systems.

[0273] In situations where the region D103 is not directly in front of the large video display D107 and the active image region D208 is deep enough that the direction of the object can be found in the direction calculation process D609, a vector may be extended from the position of the object to the video display D107 using the angle of direction to find the location on the video display where the user is "pointing."

[0274] However, the active image region D208 is often not deep enough to accurately calculate the direction in processing block D609. In these situations, when the region D103 is not immediately in front of the large video display D107 and its direction cannot be calculated, Equation 10 may be applied, where the subregion D1103 is smaller than the video display. The processor then maps the absolute position of the object D105 to the position indicator, so that movement of the object D105 approximates large movements in the position of the position indicator on the video display, thereby making the entire area of ​​the video display easily reachable by the user (e.g., the subregion D1103 may be defined to be at most 750 mm wide and correspondingly tall, a size easily reachable by many users). In this configuration, the system still provides the user with the sensation of "pointing at the screen."

[0275] In another variation of this form of user interface, the user moves the display of an indicator within a display of a three-dimensional virtual environment (examples are shown in Figures 26A and 26B). The virtual environment may be generated using a projection transform so that the depth of the virtual environment is indicated by the image shown on the video display D107. Techniques for generating this type of virtual environment include OpenGL. Equation 10 is used to remap the x, y, and z coordinates (so that the subregion 1103 becomes, for example, a cube).

[0276] Applications controlled by movable objects on screen indicators (e.g., FIGS. 25A, 26A, and 26B), the controls of which are described above, typically show graphical representations of data or interactive elements (e.g., button D1109 or object representation D1202). The user is expected to place the indicator D1101 over one of these objects, or, if a three-dimensional virtual environment is presented, touch or interact with the object. For two-dimensional interfaces, this condition may be detected by comparing the remapped indicator position D1106 with the bounds of the object's graphical representation (e.g., D1110); this condition is true if the indicator position is within the object bounds. For three-dimensional interfaces, this condition may be detected by comparing the bounds D1203 of the entire indicator D1101, or, if finer control is required, a portion of the indicator with the bounds D1204 of the object D1202. The user optionally receives feedback indicating that the cursor is placed over the object. The feedback may be in various forms, including audio cues and / or changes in the graphical representation of one or both of the cursor and object. The user may then activate, manipulate, or move the object under the cursor. By performing a gesture, the user is to indicate their intent to activate, manipulate, or move the object.

[0277] Optionally, the motion of the target object D105 may be interpreted and classified by a gesture detection module D315, as described above with respect to FIG. 17. The gesture detection process D315 may utilize data formed from any component of the system. The final coordinates D314, the image coordinates D310 and D311, or a combination of these D310, D311, and D314, may be sampled over time and provided as input to the gesture detection process D315. Using this data as input to the gesture detection process D315, various gestures (e.g., "hovering" and "poking") have been successfully detected.

[0278] In a scenario where the state of the application (i.e., whether the indicator D1101 is over the button D1109 or not) is known and communicated to the gesture detection module D315, one gesture that a user can perform to indicate an intent to activate an object (e.g., screen object D1109, D1202) under the cursor D1101 is to hover the cursor over the object (e.g., D1109, D1202) for longer than a predetermined duration. If the application state does not change for the predetermined duration, this gesture performed by the user is detected by monitoring the application state and triggering a gesture. It is not necessary to develop an application specifically for the multi-camera control system D100 because technology exists that can unobtrusively monitor the application state (by setting a "hook" using the Windows SDK function "SetWindowsHookEx" in the Windows operating system) and emulate a mouse "click" (using the Windows SDK function "SendInput" in the Windows operating system).

[0279] In some scenarios, the state of the application may not be available and may not be monitored. In this case, some example gestures that indicate an intent to activate an object (e.g., screen object D1109, D1202) under the cursor D1101 are holding the hand still ("hovering") or poking the hand back and forth rapidly.

[0280] The method for detecting "hovering" is by maintaining a history of the target object D105's position, which includes a record of all positions and states over a predetermined duration, ending with the most recent sample. This duration represents the minimum duration the user must hold their hand still. Separate minimum and maximum positions in each of the three dimensions (x, y, z) are found in the history. If the target object D105 is present within the target region D103 in all samples of the history, and the distance between the minimum and maximum is within a predetermined threshold in each of the three dimensions, a "hovering" gesture is reported. These distance thresholds represent the maximum amount the target object D105 can move, as well as the maximum amount of variation (or "jitter") that is to be introduced into the hand position by various components of the system. If the system emulates a mouse, as described above, a common way this gesture is reported is by emulating a mouse "click." Gestures representing additional mouse operations, "double-click" and "drag," are also detected and emulated.

[0281] Optionally, gestures that are independent of the position of the indicator relative to the object may additionally be detected and given meaning by the application, which may or may not depend on the state of the application. Applications using this style of interaction generally do not explicitly use or display the target object's position D317 or other positions. These applications may be controlled entirely or primarily by the system's interpretation of position alone. These applications also do not need to be developed specifically for the system, because the interpretations made by the system can be used to simulate actions that may be performed on a conventional user input device (e.g., a keyboard or joystick).

[0282] Many useful interpretations depend directly on the absolute position of the target object D105 within the target region D103 (alternatively, the indicator position D1105 within the subregion D1103 may be used in an equivalent manner). One way to make these interpretations is to define a box, plane, or other shape. If the target object D105's position (e.g., the position defined by block D314 or the position defined by the remapped coordinates from the remapping process D317) is found to be within the first box (or beyond the boundary defined by the first plane) and was not found in the previous observation (because it was elsewhere in the target region D103 or was not detected), a state is triggered ON. This state is maintained until the hand position is not found within the second box (or beyond the boundary defined by the second plane), at which point the state is triggered OFF. The second box must contain the entire first box and is generally larger. Using a larger box reduces the occurrence of accidental on and off triggers when the target object D105 is detected near the box boundary, and very little motion or minor noise in the image signal causes the position D317 to drift in and out of the box. Depending on the gesture's application, one of three ways of interpreting this state is commonly used. In one way, the gesture directly reflects the state with on and off triggers. When emulating a keyboard key or joystick fire button, it is "pressed" when the state is triggered on and "released" when the state is triggered off. In another way, the gesture is triggered only by the transition from off to on. When emulating a keyboard key or joystick button, the key is "clicked." The duration and off state are not reported to the application but are maintained so that the gesture is not repeated until the state is triggered off. As a result, each instance of a gesture requires a clearly defined intent by the user.A third method is to trigger the gesture with a transition from an on to an off state, and periodically retrigger the gesture at a predefined interval as long as the state remains on, emulating the way holding down a keyboard causes characters to repeat in some applications.

[0283] One way in which a box or plane may be defined within the region of interest D103 for the above techniques is as follows: By defining a first plane (D1501 in FIG. 27A ) and a second plane D1502 that divide the region of interest into a “fire” region D1503 and a “neutral” region D1504 (as noted above, the gesture reported when the object of interest D105 is within the region D1505 between the planes depends on the object's previous position), the above techniques can detect the object of interest D105 (typically a hand) “pushing” forward, which is a gesture to emulate a fire button on a joystick or to cause applications to react in general in conjunction with joystick button presses (e.g., firing a weapon in a video game).

[0284] Another technique for defining a box or plane within the target area D103 for the above technique is as follows. As shown in FIG. 27B, the left, right, top, and bottom portions of the target area D103 that overlap at the corners are separated to define first-type planes D1506, D1507, D1508, and D1509. Second-type planes are labeled D1510, D1511, D1512, and D1513. Each pair of first and second planes is processed independently. This combination of planes emulates four directional cursor keys. Here, a hand at the corner triggers two keys that are commonly interpreted by many applications as four secondary 45-degree (diagonal) directions. By emulating a keyboard cursor in this manner, various existing applications can be controlled by the system D100. Applications (including Microsoft® PowerPoint®) respond to emulated cursor keys (eg, up and down arrow keys) by, for example, advancing to the next or previous slide in the presentation sequence.

[0285] Another way to emulate unobtrusive directional control applies to applications that expect four 45-degree directional states to be explicitly represented. As shown in FIG. 27C, boxes D1514, D1515, D1516, and D1517 are defined for each of the four primary (horizontal and vertical) directions, and additional boxes D1518, D1519, D1520, and D1521 are defined for each of the secondary 45-degree (diagonal) directions. For clarity, only the first type of box is shown. Gaps are placed between these boxes. FIG. 27D illustrates how adjacent boxes are defined. The gap between the first type of boxes D1522 and D1523 ensures that the user intentionally places the target object D105 in the box, while the gap D1524 is filled by partially overlapping second type of boxes D1525 and D1526. As a result, the system reports the previous gesture until the user explicitly intends to move the target object D105 to an adjacent box or to the central neutral area. This combination of buttons can be used to emulate an eight-way joystick pad.

[0286] A more widespread type of gesture relies on motion instead of or in addition to position. One example is the "swipe hand left" gesture. This is a gesture that communicates to an application that you want to return to the previous page or state. Through keyboard and mouse emulation, this gesture may be used to control information presentation software (particularly Microsoft® PowerPoint®) to go to the previous page in a presentation sequence. Through keyboard and mouse emulation, this gesture causes a web browser to perform the action associated with a "back" button. Similarly, the "swipe hand right" gesture is a gesture that communicates to an application that the user wants to proceed to the next page or state. For example, this gesture causes presentation software to advance to the next slide in a presentation sequence and browser software to advance to the next page.

[0287] One method for detecting a "hand swipe left" is as follows: A thin stripe along the leftmost portion of the target region D103 is defined as the leftmost region. The position of the target object D105 (e.g., the position defined by block D314 or the position defined by the remapped coordinates from the remapping process D317) is represented as three states: 1. The target object is present and not inside the leftmost region. 2. The target object is present and inside the leftmost region. 3. The target object is not within the hand detection area.

[0288] The transition from State 1 to State 2 above causes the gesture detection module D315 to enter a state where it starts a timer and waits for the next transition. If a transition to State 3 is observed within a predetermined duration, a "hand swipe left" gesture is reported. This technique is generally replicated for right, top, and bottom edges, and also for "hand pull back" since the hand position is found in three dimensions.

[0289] Various gesture detection techniques have been described. Still other gesture detection techniques (e.g., hidden Markov layers) have been described in the research literature and may be applied in various implementations of system D100 described herein.

[0290] Referring again to Figures 15 and 17, another embodiment of the multi-camera control system D100 will be described in further detail. While Figure 15 shows a two-camera system, it should be understood that the image processor D106 can be configured to receive input from more than two cameras and may include four or more video cameras for particular applications. In a four-camera embodiment, components D304-D311 of Figure 17 are replicated to support two additional cameras. Additionally, the combination module D312 is configured to receive presence and position information (similar to data D310 and D311) associated with four sets of cameras associated with the tracked target object D105. The techniques and equations described above (especially Equation 5 and Equation 6) can be applied to additional camera pairs. Here, the output of the combination module D312 is the average of all positions from each of the camera pairs. The gesture detection module D315 is similarly reconfigured to receive presence and position information D310, D311 relating to the four sets of cameras from two further detection modules (similar to D308, D309) that are substantially similar to the detection modules D310 and D311.

[0291] The output from the image processor 106 (which in this case includes processed object position coordinates and gesture information associated with the four cameras) can be used by another process or by a user application program 316. The formulas and geometry (described above) used to calculate coordinate information associated with the target object 105 from the two further cameras are also used.

[0292] In one four-camera embodiment, two additional cameras are positioned at the bottom two corners of the controlled background D104 and oriented so that the target region D103 is within each camera's field of view D205. An advantage of a four-camera system is that the position of the target object D105 can be tracked with greater accuracy. Thus, an application program may include more screen objects and a higher density on the video display D107, because improved tracking accuracy allows very close objects to be correctly selected with small movements of the target object D105. Furthermore, the two additional cameras reduce errors in tracking the target object D105 when portions of the target object D105 are occluded in the field of view D205 associated with one or more other cameras.

[0293] (Device neutral position) According to one general aspect, a method is disclosed that includes determining a neutral position of a device about at least a first axis and measuring an angular displacement of the device about at least the first axis, the device including a first control associated with at least a first plurality of output signals, and also includes receiving a selection of the first control and outputting one of the first plurality of output signals based on at least the selection and the angular displacement.

[0294] Embodiments may include one or more of the following features. For example, a neutral position of the device may be determined with respect to at least a second axis (orthogonal to the first axis), where the angular displacement may include a first axis component and a second axis component. Further, a neutral position of the device may be determined with respect to at least a third axis (orthogonal to the first and second axes), where the angular displacement may include a third axis component. The first axis, the second axis, and / or the third axis may intersect within the device.

[0295] The first control may be associated with at least three output signals or at least nine output signals, where each of the plurality of output signals may correspond to a character, such as an alphanumeric character. The method may further include displaying the output signals and / or displaying an indication of the angular displacement. The method may also further include defining a plurality of tilt regions about a first axis, where one of the first plurality of output signals is an output based on the plurality of tilt regions. Angular displacement of the device about the first axis may be measured as 0 degrees, and the first tilt region includes an angular displacement of 0 degrees. Alternatively, the first tilt region may be defined as a region including approximately -30 degrees to 0 degrees about the first axis, and the second tilt region may be defined as a region including approximately 0 degrees to +30 degrees about the first axis. In a further aspect, if the angular displacement is within the first tilt region when the selection is received, the first output signal may be output. If the angular displacement is within the second tilt region when the selection is received, the second output signal may be output. A third or fourth output signal may be output if the angular displacement is within the third or fourth tilt region, respectively, when the selection is received.

[0296] The method may also define a plurality of first-axis tilt regions about the first axis and a plurality of second-axis tilt regions about the second axis. Here, one of the first plurality of output signals may also be output based on the plurality of first-axis tilt regions and / or the plurality of second-axis tilt regions. When the selection is received, a first output signal may be output if the first axis component is in the first first-axis tilt region and the second axis component is in the first second-axis tilt region. A second output signal may be output if the first axis component is in the second first-axis tilt region and the second axis component is in the first second-axis tilt region. A third output signal may be output if the first axis component is in the second first-axis tilt region and the second axis component is in the second second-axis tilt region. And / or a fourth output signal may be output if the first axis component is in the second first-axis tilt region and the second axis component is in the second second-axis tilt region.

[0297] Alternatively, in another aspect, when a selection is received, a first output signal may be output if the first component is in the first first-axis tilt region and the second axis component is in the first second-axis tilt region. A second output signal may be output if the first component is in the first first-axis tilt region and the second axis component is in the second second-axis tilt region. A third output signal may be output if the first component is in the first first-axis tilt region and the second axis component is in the third second-axis tilt region. A fourth output signal may be output if the first component is in the second first-axis tilt region and the second axis component is in the first second-axis tilt region. A fifth output signal may be output if the first component is in the second first-axis tilt region and the second axis component is in the second second-axis tilt region. A sixth output signal may be output if the first component is in the second first-axis tilt region and the second axis component is in the third second-axis tilt region. A seventh output signal may be output when the first component is in the third first-axis tilt region and the second axis component is in the first second-axis tilt region, an eighth output signal may be output when the first component is in the third first-axis tilt region and the second axis component is in the second second-axis tilt region, and / or a ninth output signal may be output when the first component is in the third first-axis tilt region and the second axis component is in the third second-axis tilt region.

[0298] According to another general aspect, an apparatus is disclosed, the apparatus including a tilt sensor configured to determine a neutral position of the apparatus about at least a first axis and further configured to measure angular displacement of the apparatus about at least the first axis, the apparatus also including at least a first control associated with a first plurality of output signals, and a processor configured to receive a selection of the first control and further configured to output one of the first plurality of output signals based on at least the selection and the angular displacement.

[0299] Implementations may include one or more of the following features. For example, the first axis and the second axis may intersect at a center of the device or at a peripheral portion of the device. The device may further include at least second through tenth controls associated with second through tenth output signals, respectively. The first control may be a button, and / or the device may be a phone. The displacement signal may be measured using a tilt sensor, and the tilt sensor may be a gyroscope. The device may further include a display configured to display the output signal and / or a display configured to display an indication of the angular displacement, and the device may further include a keyboard configured to input a selection.

[0300] According to another general aspect, a computer program product tangibly stored on a computer-readable medium is disclosed. The computer program product is operable to cause a computer to perform operations including determining a neutral position of a device about at least a first axis and measuring an angular displacement of the device about at least the first axis, the device including a first control associated with at least a first plurality of output signals. The computer program product is also operable to cause the computer to perform operations including receiving a selection of the first control and outputting one of the first plurality of output signals based at least on the selection and the angular displacement.

[0301] According to another general aspect, a telephone device is disclosed. The telephone device includes a tilt sensor configured to determine a neutral position of the telephone device about at least a roll axis and further configured to measure angular displacement of the telephone device about at least the roll axis. The telephone device also includes at least first through eighth buttons respectively associated with at least four alphanumeric characters. The telephone device further includes a processor configured to receive a selection of the first button and further configured to output one of the at least four alphanumeric characters based on at least the selection and the angular displacement.

[0302] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

[0303] 28 shows an exterior view of a device according to one exemplary embodiment, with the device in a neutral position. The hardware environment of device E100 includes a keypad including at least a first control E102 for inputting text data and user commands into device E100, a display E105 for displaying text and images to the user, and an indicator (e.g., tilt indicator E106) for displaying an indication of angular displacement or tilt orientation about at least one axis.

[0304] Display E105 displays graphics, images, and text and includes a user interface for software applications used by this embodiment, as well as the operating system programs necessary to operate device E100. A user of device E100 uses first control E102 to input commands and data to operate and control the operating system programs and application programs.

[0305] Display E105 is configured to display a GUI to a user of device E100. A speaker may be present and may generate voice and audio data received from application programs running on device E100 (e.g., voice from another user generated by a telephony application program) or ring tones generated by a ring tone application program. A microphone may be used to capture audio data generated by the user, for example, when the user is making a call with another user via device E100. Additionally, tilt indicator E106 is configured to indicate the angular displacement or tilt orientation of device E100 and to provide visual feedback to a user of device E100, informing the user of the tilt orientation, which may be used to interpret control selections.

[0306] Operation of device E100 is based on the orientation of the device in two states: a "neutral" position and a "selected" position corresponding to the position of the device before, concurrently with, or after the selection of first control E102. More specifically, as explained more fully below, the output signal by device E100 depends on the angular displacement between the neutral position and the selected position about at least one axis, where the angular displacement has an angular displacement component for each axis of interest.

[0307] 28, for example, shows device E100 in one possible three-axis neutral position. In particular, orthogonal X, Y, and Z axes intersect at the center of device E100, with the X axis extending parallel to the longitudinal axis of device E100. According to this exemplary neutral position, rotation about the X axis achieves a rolling motion, rotation about the Y axis achieves a pitching motion, and rotation about the Z axis achieves a yawing motion. These rolling, pitching, and yawing motions are generally referred to herein as "tilting" motions.

[0308] The determination of the number of axes of interest, and the position and orientation of the axes relative to device E100, are device- and application-specific, and no limitations to these aspects are inferred in the following description. For example, if it is undesirable or impossible to operate the device in a yawing motion, or if motion about one or two axes can be used to effectively control the number of output signals, the neutral position of the device may be determined with respect to only these one or two axes. Furthermore, at least one axis may not intersect device E100, or at least one axis may extend along a portion of the periphery or edge of device E100. In addition, one of the axes may extend parallel to the longitudinal direction of device E100 or at an angle relative to the longitudinal direction of device E100. In either case, the neutral position is aligned with an axis relative to the Earth (e.g., magnetic north or true north, or an axis pointing toward the center of the Earth or the horizon), or with respect to the user, device, or other axis.

[0309] For telephony, a 1-axis neutral position is provided when angular displacement is measured with respect to rolling rotation about the X axis, and a 2-axis neutral position is provided when angular displacement is measured with respect to rolling and pitching rotation about the X and Y axes, respectively. In either case, the X and Y axes intersect at the center of the device, with the X axis extending in a longitudinal direction parallel to the longitudinal direction of the device. Other neutral position orientations are also contemplated.

[0310] When entering text into a device such as a phone, a user typically holds the device at a positive (upward) pitch angle while looking at the display. In that regard, the phone's X-axis in the neutral position may be defined as a similar upward angle, so that flattening the phone angle relative to the ground may register as a pitched forward motion. In other cases, of course, an X-axis parallel to the ground is the "neutral" X-axis position.

[0311] In FIG. 28, device E100 is illustrated as a mobile phone, but in further embodiments, device E100 may include a desktop PC, a laptop, a workstation, a mid-range computer, a mainframe computer, a handheld computer, a tablet computer, a personal digital assistant ("PDA"), or another type of embedded system (e.g., a computer keyboard or remote control).

[0312] Figure 29 shows an example of the internal architecture of the embodiment of Figure 28. The computing environment includes a processor E200 on which computer instructions, including an operating system or applications, are processed, a display interface E202 providing a communication interface and processing functionality for generating graphics, images, and text on a display E105, a keypad interface E204 providing a communication interface to a keypad including a first control E102, a tilt sensor E206 for measuring angular displacement of the device E100 about at least a first axis, an indicator interface E208 providing a communication interface to an indicator including a tilt indicator E106, a random access memory ("RAM") E210 on which computer instructions and data for processing by the processor E200 are stored in a volatile memory device, and a processor E210 for performing basic system functions. The system includes a read-only memory ("ROM") E211 in which unchanging low-level system code or data for the system (e.g., basic input / output ("I / O"), startup, or receipt of keystrokes from a keypad) is stored in a non-volatile memory device, and optionally a memory E220 or other suitable type of memory (e.g., random access memory ("RAM"), read-only memory ("ROM"), programmable read-only memory ("PROM"), erasable PROM ("EPROM"), electrically erasable PROM ("EEPROM"), magnetic disk, optical disk, floppy disk, hard disk, removable cartridge, flash drive) in which files including an operating system E230, application programs E240, and data files E246 are stored. The component devices and processor E200 communicate with each other via a bus E250.

[0313] The RAM E210 interfaces with the bus E250 to provide rapid RAM storage for the processor E200 during execution of software programs, such as an operating system, application programs, and device drivers. More specifically, to execute a software program, the processor E200 loads computer-executable processes from a memory medium into a field of the RAM E210. Data is stored in the RAM E210, and the data is accessed by the processor E200 during execution.

[0314] 29, storage device E220 stores computer-executable code for operating system E230, application programs E240 (e.g., word processors, spreadsheets, presentations, graphics, image interpretation training, games, or other applications), and data files E246. While the above implementations can be used, it is also possible to implement functionality according to the present disclosure as a dynamic link library ("DLL") or as a plug-in to other application programs, such as an Internet web browser (e.g., the MICROSOFT® Internet Explorer web browser).

[0315] Processor E200 is one of many high performance computer processors, including an INTEL® or AMD® processor, a POWER PC® processor, a MIPS® reduced instruction set computer ("RISC") processor, a SPARC® processor, an HP ALPHASERVER® processor, an ACORN® RISC Machine ("ARM®") architecture processor, or a proprietary computer processor for a computer or embedded system without departing from the scope of this disclosure. In additional arrangements, processor E200 of apparatus E100 is a plurality of processing units, including multiple CPU configurations found in high performance workstations and servers, or multiple scalable processing units found in mainframes.

[0316] The operating system E230 may be a MICROSOFT® WINDOWS NT® / WINDOWS® 2000 / WINDOWS® XP workstation, a WINDOWS NT® / WINDOWS® 2000 / WINDOWS® XP server, various UNIX®-based operating systems (including AIX® for IBM® workstations and servers, SUNOS® for SUN® workstations and servers, LINUX® for INTEL® CPU-based workstations and servers, HP UX WORKLOAD MANAGER® for HP® workstations and servers, IRIX® for SGI® workstations and servers, VAX / VMS for Digital Equipment Corporation computers, OPENVMS® for HP® ALPHASERVER-based computers, MAC OS® X for POWERPC®-based workstations and servers), SYMBIAN OS® for mobile devices, WINDOWS The operating system E230 may be MOBILE® or WINDOWS CE®, PALM®, NOKIA® OS ("NOS"), OSE® or EPOC®, or any proprietary operating system for a computer or embedded system. The application development platform or framework for the operating system E230 may be BINARY RUNTIME ENVIRONMENT FOR WIRELESS® ("BREW®"), Java Platform Micro Edition ("Java ME") or Java 2 Platform Micro Edition ("J2ME®"), PYTHON®, FLASH LITE®, or MICROSOFT® NET Compact.

[0317] The tilt sensor E206, as described below, detects the orientation of the device E100 and may be a gyroscope, optical sensor, and / or other type of tilt sensor. The optical sensor may be used to detect the orientation of the device E100 and determine the motion and orientation of the device E100, for example, using the optical flow of a series of images from a camera incorporated into the device E100. The optical flow indicates the apparent relative velocity of features within the series of images. Because the optical flow is relative to the camera, the motion of the camera provides the apparent velocity of features within the camera's field of view. The camera's motion is calculated from the apparent velocity of features within the camera's field of view. The position or orientation may also be calculated relative to a neutral position over time. While the tilt sensor E206 has been described as an optical sensor using an optical flow method to track the tilt or slope of the device E100 using a camera, in other embodiments, the tilt or slope of the device E100 is tracked without using the optical flow method, for example, using an accelerometer.

[0318] The computer-readable memory medium stores information within device E100 and may be volatile or nonvolatile. The memory may be capable of providing mass storage for device E100. In various different implementations, the memory may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive. Figures 28 and 29 illustrate one possible implementation of a computing system for executing program code or program steps or process steps, although other types of computers or devices may be used.

[0319] 30 is a flowchart illustrating a method according to another exemplary embodiment. Briefly, the method includes determining a neutral position of a device about at least a first axis and measuring angular displacement of the device about at least the first axis, the device including a first control associated with at least a first plurality of output signals. The method also includes receiving a selection of the first control and outputting one of the first plurality of output signals based on at least the selection and the angular displacement.

[0320] More specifically, method E300 begins (step ES301) and defines a plurality of tilt regions about a first axis (step ES302). As described in more detail below, the output of an output signal is based at least on the angular displacement of the device at the time of selection of a first control. According to one aspect, tilt "regions" are defined such that, at the time of selection of the control, if the angular displacement is within a particular tilt region or band of angles, then the output associated with the tilt region is output.

[0321] 31A-31D illustrate some example tilt regions for a hypothetical neutral axis labeled "n-axis," where neutral represents the neutral X-axis, Y-axis, and / or Z-axis. Each of the X-axis, Y-axis, or Z-axis can have an individually determined tilt region. A common tilt region definition can be applied to multiple axes. Alternatively, an axis may not have a defined tilt region.

[0322] 31A illustrates two example tilt regions defined about the neutral axis. An angular displacement of about -90 degrees to 0 degrees about the neutral axis is in region E401, and an angular displacement of about 0 degrees to about 90 degrees about the neutral axis is in region E402. An angular displacement of about 91 degrees to -91 degrees (indicating the device is upside down) does not correspond to either region, and an angular displacement of exactly 0 degrees is in either region E401 or region E402.

[0323] If the neutral axis represents the X-axis, angular displacement in region E401 may be due to a negative (left) roll of the device, and angular displacement in region E402 may be due to a positive (right) roll of the device. If the neutral axis represents the Y-axis, angular displacement in region E401 may be due to a negative (forward) pitch of the device, and angular displacement in region E402 may be due to a positive (backward) pitch of the device. If the neutral axis represents the Z-axis, angular displacement in region E401 may be due to a negative (counterclockwise) yaw, and angular displacement in region E402 may be due to a positive (clockwise) yaw. Although two tilt regions are shown, any number of tilt regions may be defined, depending largely on the sensitivity of the tilt sensor, the number of output signals associated with each control, and the user's ability to distinguish small angles when operating the device.

[0324] In either case, the signal output by the device depends on the angular displacement and tilt range. For example, the device may output a first of a plurality of signals when the angular displacement of the device is within a first range and a second of a plurality of signals when the angular displacement of the device is within a second range, even if the same control is selected in both situations. While FIG. 28 illustrates ranges E401 and E402 as including a ±90 degree band, in a similar embodiment, tilt range E401 defines a range including approximately −30 degrees to 0 degrees relative to the neutral axis, and tilt range E402 defines a range including approximately 0 degrees to +30 degrees relative to the neutral axis.

[0325] FIG. 31B shows four tilt zones defined about the neutral axis, with a dead space between the zones at 0 degrees about the neutral axis. It is often desirable to define a dead space between two distinct adjacent zones because the tilt sensor is insensitive, the user cannot distinguish, or for various other reasons. If the neutral axis represents the Y axis, an angular offset between approximately 91 degrees and -91 degrees (representing an upside-down device) or an angular offset of approximately 0 degrees does not correspond to a tilt zone. If a control is selected when the device is not pointed at a tilt zone, the default output is output, the last output is output, no output is produced, the output associated with the nearest or complementary tilt zone is output, or another type of output is output.

[0326] The angular deviation in region E404 results from the device's strong negative gradient, but the angular deviation in region E405 also results from a less negative gradient than the negative gradient of region E404. The tilt deviation in region E407 results from a strong positive gradient, but the angular deviation in region E406 also results from a less positive gradient than the negative gradient of region E407.

[0327] FIG. 31C shows an example of two tilt regions defined about a neutral axis, where the region around the neutral axis is substantially the first region, approximately 0 degrees. In particular, if the neutral axis represents the X axis, the device will remain in region E409 if rotated negatively, not moved from the neutral position, or gently rotated in a positive direction. A strong positive rotation must occur to orient the device toward region E410. The tilt region shown in FIG. 31C is desirable, e.g., where region E409 represents a default desired output, and aggressive, high-amplitude manipulation of the device is required to position the device in region E410, thereby overriding the default desired output. In the example of FIG. 31C, tilt region E409 includes a 0-degree angular misalignment, where the device's angular misalignment about the first axis is measured at 0 degrees and is within tilt region E409.

[0328] FIG. 31D shows an example of two tilt regions defined about the neutral axis, where a single region occupies a band of angular misalignment on either side of the neutral axis. More specifically, region E412 is defined by the region surrounding 0 degrees about the neutral axis, and region E411 occupies symmetrical angular bands in positive and negative angular directions. If the neutral axis represents the Z axis, the angular misalignment in region E411 results from a high amplitude positive or negative yaw. The angular misalignment in region E412 results from a more moderate positive or negative yaw, or from the orientation of the device in the neutral position.

[0329] In any of the above examples, the neutral axis may represent the X, Y, and / or Z axis, thus effectively increasing the total number of available tilt regions. For example, if the neutral axis in FIG. 31A represents the X axis and the neutral axis in FIG. 31B represents the Y axis, a total of eight tilt regions are available, because the four gradient tilt regions in FIG. 31B are each divided into two rotational tilt regions in the example of FIG. 31A. Assuming each axis has an equal number, n, of tilt regions, the total number of tilt regions for a two-axis configuration is n2, and the total number of tilt regions for a three-axis configuration is n3.

[0330] Finally, in some instances, it is not necessary to define a tilt range, since the angular deviation, not the tilt range, itself determines the output signal. Furthermore, a tilt range can also potentially be defined if the range of motion about a desired axis is equally divisible by the number of output signals, where each output signal corresponds to a mathematically determined angular range.

[0331] Returning to FIG. 30, a neutral position of a device is determined relative to at least a first axis, the device including at least a first control associated with a first plurality of output signals (step ES304).

[0332] FIG. 32 shows a top exterior view of an example device according to another exemplary implementation. Device E500, i.e., a mobile phone, has a keypad including at least a first control E502 associated with a first plurality of output signals. In the illustrated example, first control E502 is a key or button on the keypad or keyboard of device E500, with each individual control representing a different alphanumeric character or symbol. In particular, first control E502 is labeled "9" and corresponds to four output signals representing the letters "W," "X," "Y," and "Z," or to twelve output signals representing the case-sensitive letters "W," "X," "Y," "Z," "w," "x," "y," and "z," as well as the symbols ","," "."," " / ," and "'.". There is no limit to the number of output signals or characters that can correspond to a single control. In certain embodiments, first control E502 is associated with multiple output signals, e.g., three output signals or nine output signals. Each of the plurality of output signals may correspond to a character, such as an alphanumeric character or symbol.

[0333] The neutral position of the device E500 is determined, for example, prior to or after the selection of the first control, or when the device E500 is powered on at the operating location. In one embodiment, a memory buffer stores tilt sensor output data, and the neutral position of the device E500 is reconstructed from the orientation of the device E500 when the control is selected and the output data. In another embodiment, the neutral position is a factory preset state, for example, where the neutral X-axis is defined as extending perpendicular to the center of the Earth and angular deviation is measured when the device E500 is facing any direction other than up. In a further embodiment, the processor, tilt sensor, and memory communicate to determine a common neutral position based on the average position of the device E500 whenever a control is normally selected. In yet a further embodiment, the neutral position is user-selectable. In any respect, the neutral position effectively acts to reset the tilt sensors to 0 degrees across each axis of interest, where any movement of device E500 away from the neutral position serves to register an angular misalignment. Relative to the user of device E500 or the Earth, the neutral position may be a flat position, a vertical position, or an oblique or tilted position.

[0334] In further embodiments, the neutral position of device E500 is determined with respect to at least a second axis, orthogonal to the first axis, where the angular offset includes a first axis component and a second axis component. In further embodiments, the neutral position of device E500 is determined with respect to at least a third axis, orthogonal to the first axis and the second axis, where the angular offset includes a third axis component. The first axis, second axis, and / or third axis intersect within device E500, outside device E500, or along a peripheral location or at an edge of device E500.

[0335] Entering text into the device is facilitated because device E500 includes a tilt sensor that detects the orientation of the device. For example, the tilt sensor detects the degree to which the device is rotated left, right, or tilted up or down, where the tilt direction or angular deviation of the device about that axis indicates how selection of control E502 is interpreted and output. For example, if control E502 corresponds to multiple characters, the orientation of device E502 identifies which of the multiple characters is output when control E502 is selected, or when the appropriate character is output.

[0336] Using the device orientation to identify the character to be output allows a character to be output each time a single control is selected, increasing the speed of text entry by reducing the number of control selections required to enter text. Because a fixed number of control selections represents the input of a character, the user may identify the next character immediately after the current character is identified, eliminating the need to wait a predetermined number of times before identifying the next character and also increasing the speed of text entry.

[0337] As noted above, the neutral position of the device is a reference orientation from which angular deviation is measured about at least one axis relative to a selected position, which corresponds to the position of the device prior to, at, or after selection of a control, such as the first control. In one embodiment, the neutral position of the device is determined relative to one axis, with the neutral position determined as a "flat" position, where one axis is parallel to the ground. In another embodiment, the neutral position of the device is determined relative to two axes, with the neutral position ergonomically determined as the orientation of the device when it is normally held by a user of the device. In a further embodiment, the neutral position of the device is determined relative to three axes, with one axis determined parallel to the magnetic north-south axis, one axis determined parallel to the east-west axis, and a third axis determined to point toward or away from the center of the Earth.

[0338] Returning to FIG. 30, the angular deviation of the device is measured about at least a first axis (step ES305). In particular, a tilt sensor, such as tilt sensor E206, measures the angular deviation between the device's current position and a neutral position, where the angular deviation includes a component for each axis of interest. In one embodiment, tilt sensor E206 measures the angular deviation of the device when a control is selected. Selection of the control itself may affect the orientation of the device; in another embodiment, the tilt sensor measures the angular deviation of the device before or after the control is selected.

[0339] The tilt sensor detects the orientation of the device. For example, the tilt sensor detects the degree to which the device is rotated left or right, tilted up or down, or yawed clockwise or counterclockwise. In one embodiment, the tilt sensor measures at least two distinct levels of rotational tilt about the X axis, where the device may be rotated left, rotated right, or neither left nor right. Additionally, the tilt sensor measures at least two distinct levels of slope tilt about the Y axis, which is the front-to-back direction, where the device may be tilted up, tilted down, or neither up nor down. Additionally, the tilt sensor measures at least two distinct levels of yaw tilt about the Z axis, where the device may be yawed clockwise or counterclockwise, or may not yaw. In such an implementation, the tilt sensor indicates that the device has rotated left when the device is rotated 1.5 to 4.5 degrees left. As another example, if the device tilts forward less than 1.5 degrees and backward less than 1.5 degrees, the tilt sensor indicates that the device is not tilting forward or backward. In another implementation, the tilt sensor may indicate four or more levels of tilt in each of the left-to-right and front-to-back directions. In such an implementation, each level of tilt in a particular direction corresponds to a range of angles at which the device is tilted.

[0340] An indication of the angular misalignment is displayed (step ES306). As mentioned above, the direction of the neutral position need not be indicated to the user. Furthermore, each axis may have two or more tilt regions in each direction about each axis. For these and other reasons, an indicator is provided to display either an indication of the angular misalignment or an indication of the tilt region to which the angular misalignment corresponds in real time or near real time. If the angular misalignment is measured before or after a control is selected, the indicator estimates the appropriate angular misalignment or tilt region indication based on all available information. If the neutral position is defined relative to two or more axes, the user can determine which axis the indicator will display; the indicator can have a default or preset axis in question, or the determination can be context-dependent.

[0341] 33A-33B illustrate examples of indicators according to one exemplary embodiment. In FIG. 33A, indicator E600 indicates the orientation of the device on the display. The indicator provides visual feedback so that the user is aware of the orientation of the device to use to interpret control selections.

[0342] Indicator E600 includes a positive tilt indicator E601 and a negative tilt indicator E604, which point in the negative (left) and positive (right) directions, respectively. In addition, indicator E600 includes a center indicator E602 that is visually distinct from the positive tilt indicator E601 and the negative tilt indicator E604 when the device is not tilted, such as when the device is in a neutral position or a position not registered by the tilt sensor (e.g., upside down). When the device is tilted in the indicated direction, one of the tilt indicators is illuminated or otherwise visually distinct from the other tilt indicator and center indicator E602. Furthermore, when the device is not being rocked left or right, center indicator E602 is illuminated or otherwise visually distinct from the positive tilt indicator E601 and the negative tilt indicator E604. The center indicator is, for example, illuminated when the device is oriented as shown in FIG. 28 . Positive tilt indicator E601 is illuminated when the device is oriented as shown in region E402 of Figure 31A, and negative tilt indicator E604 is illuminated when the device is oriented as shown in region E401 of Figure 31A.

[0343] In another embodiment illustrated in FIGS. 33B and 33C, indicator E605 also includes two partial tilt indicators E606 and E607, which also point in the negative and positive directions, respectively. Each of the partial tilt indicators is located between center indicator E604 and either negative tilt indicator E604 or positive tilt indicator E601. The partial tilt indicators are illuminated or otherwise visually distinguished from other components of indicator E605 when the device is partially tilted in the indicated direction. In one embodiment, both the partial tilt indicator and center indicator are illuminated when the device is partially tilted in the corresponding direction. For example, when the device is oriented in tilt region E404 of FIG. 31B, negative tilt indicator E604 is illuminated. When the device is oriented in tilt region E405 of FIG. 31B, negative tilt indicator E606 and center indicator E602 are illuminated. When the device is oriented in the neutral position as shown in FIG. 28, the center indicator 602 is illuminated. When the device is oriented in tilt region E406 of FIG. 31B, the partial positive tilt indicator E607 and the center indicator 602 are illuminated. And when the device is oriented in tilt region E407 of FIG. 31B, the positive tilt indicator E601 is illuminated. Any number of tilt indicators or partial tilt indicators are contemplated for each axis. For axes having dozens of associated tilt regions, for example, the same, more, or fewer tilt indicators may be used to provide visual feedback.

[0344] FIG. 33D illustrates a two-axis tilt indicator that may be presented on a display. Although the axes described in connection with FIG. 33D are referred to as pitch (forward / backward) and roll (left / right), these designations are arbitrary, and a set of indicators is also possible for the yaw axis or other axes. Indicator E609 operates similarly to indicator E605 with respect to one axis. However, indicator E609 also integrates a pitch tilt indicator consisting of a negative pitch indicator E610, a partial negative pitch indicator E611, a partial positive pitch indicator E612, and a positive pitch indicator E614, as opposed to the previously described one-axis indicator E605 (which was described as a roll indicator). In another embodiment illustrated in FIG. 33E, the indicator includes a single feature E615 that indicates the importance of the device's orientation. For example, the single feature indicator indicates whether a number may be output for measuring the device's angular displacement.

[0345] 28 and 33 as a series of arrows or intuitive lights, in one embodiment the indicator is incorporated into a display (e.g., display E105), or the indicator is a speaker that generates a sound or sound file that represents the tilt of the device to the user via audio. Furthermore, in another embodiment, the angular displacement or tilt area is not displayed or otherwise generated.

[0346] Returning briefly to FIG. 30, a selection of a first control is received (step ES307). In one aspect, the control is a keypad button, and selection occurs when the user presses the button. This allows a signal indicating that selection of the keypad button has occurred to be generated and sent to the processor. In another aspect, the control is not a physical control, but rather an icon on the touchscreen. In this aspect, selection occurs when the user touches an area of ​​the touchscreen associated with the icon. Here, the touchscreen application reads the coordinates of the touch, relates the coordinates to the location of the icon, and sends a signal indicating that the control has been selected. Selection of other types of controls is also contemplated.

[0347] According to the embodiment of FIG. 32, device E500 includes a keypad, or grouping of controls, with which a user can input text to interact with a GUI presented on display E505. Each control corresponds to multiple output signals (each output signal associated with a letter). In one embodiment, the keypad includes eight controls labeled "2" through "9," each corresponding to multiple letters and numbers. For example, the control labeled "2" corresponds to the letters "A," "B," and "C." In addition, other controls included on the keypad perform other text input functions. For example, the control labeled "*" is used to change the case of the next character to be output. The control labeled "0" is used to advance to the next character after the current character is characterized, and the control labeled "#" is used to insert the character "space."

[0348] One of the first plurality of output signals is output based on at least the selection and angular displacement (step ES309), or at least the selection, angular displacement, and the plurality of tilt regions. Since a first control is associated with the first plurality of output signals, the angular displacement or the angular displacement and the plurality of tilt regions is used to determine which one of the first plurality of output signals is output. In one embodiment, the neutral position of the device is determined with respect to one axis, where three tilt regions are defined about that one axis, and the first control is associated with the three tilt regions. In this case, if the angular displacement is in the first tilt region, the first output signal is output. If the angular displacement is in the second tilt region, the second output signal is output, and if the angular displacement is in the third tilt region, the third output signal is output. In another embodiment, the output signal is output based on the angular displacement and the number of output signals associated with the first control based on a formula or algorithm.

[0349] Various figures represent front and side views of the device of FIG. 32 in different states of operation. In particular, FIGS. 34A and 34B illustrate front and side views, respectively, of device E500 in a neutral position. FIG. 35A illustrates a front view of the device operated to a negative roll about the X axis, and FIG. 35B illustrates a front view of the device operated to a positive roll about the X axis. Similarly, FIG. 36A illustrates a side view of the device operated to a positive pitch about the Y axis, and FIG. 36B illustrates a side view of the device operated to a negative pitch about the Y axis. In FIGS. 35 and 36, the device is tilted approximately + / - 30 degrees about the respective axis from the neutral position shown in FIG. 34.

[0350] The orientation of the device, as indicated by the angular displacement measured by the tilt sensor, affects the output signal produced by the device when a keypad control is selected, affecting, for example, the character generated by selecting the control. Each of the multiple characters or output signals represented by a single keypad control corresponds to a different orientation of the device. When one of the keypad controls is selected, the device identifies multiple characters corresponding to the selected control and the orientation of the device indicated by the tilt sensor. One of the multiple characters and a case for the character is identified based on the identified orientation, and the identified character is output.

[0351] The degree to which the device is rolled left or right when a control is selected affects which one of the multiple characters represented by the control is output. In one embodiment, a control representing multiple characters represents three characters, and the characters represented by the control are aligned left to right on the control. The device is configured to indicate whether the device is rolled left, right, or not rolled left or right. In one such embodiment, rolling the device left when the control is selected indicates that the left-most aligned character should be output. Similarly, rolling the device right when the control is selected indicates that the right-most aligned character should be output. Finally, holding the device in a neutral position when the control is selected indicates that the center character should be output.

[0352] In another embodiment, rolling the device left when the control is selected indicates that the right-most aligned character should be output. Rolling the device right when the control is selected indicates that the left-most aligned character should be output. Holding the device in a neutral position when the control is selected indicates that the center character should be output. For example, this type of embodiment may be used because rolling the device left causes the right-most aligned character to appear at the top and more prominent than the other characters, and rolling the device right causes the left-most aligned character to appear at the top and more prominent than the other characters.

[0353] In other embodiments, the keypad controls represent more than three characters (e.g., three letters and a number, or four letters and a number). For example, the "7" control on a conventional telephone corresponds to the letters "P," "Q," "R," and "S" and the number "7." In such cases, the tilt sensor is configured to identify three or more distinct left and right roll positions, such that one of the three or more characters represented by the selected control can be identified based solely on the roll direction of the device. Each of the distinct roll positions corresponds to one of the characters represented by the selected control. For example, if the selected control is the "7" key, rolling the device as shown in region E404 of FIG. 31B indicates that the letter "P" should be output. Rolling the device as shown in region E405 of FIG. 31B indicates that the letter "Q" should be output. Rolling the device as shown in region E406 of FIG. 31B indicates that the letter "R" should be output. Rolling the device as shown in region E407 of FIG. 31B indicates that the letter "S" should be output. And the device oriented in the neutral position as shown in Figure 28 indicates that the number "7" should be output.

[0354] The roll direction of the device is used to identify the character to be output, while the pitch direction of the device is used to identify the case for that character. In one embodiment, a device that is pitched (or tilted) forward when the control is selected causes the character identified by the roll (tilt left or right) direction of the device to be output in uppercase. Similarly, a device that is not pitched forward or backward (in a neutral pitch position) when the control is selected causes the character identified by the roll (tilt left or right) direction of the device to be output in lowercase.

[0355] In some embodiments, pitching (or tilting) the device backward may cause a symbol to be output. This symbol may be a symbol that corresponds to the number represented by the selected control on a conventional computer keyboard. For example, if a control representing the number "1" is selected when the device is tilted backward, the symbol "!" may be output because the symbol "!" corresponds to the number "1" on a conventional computer keyboard (e.g., pressing "shift" and "1" on a computer keyboard outputs the letter "!").

[0356] The tilt sensor is capable of detecting tilt positions in more pitch directions than are necessary to indicate the case of the character being output. Thus, pitch positions not used to indicate the case of a character may be used to select a character. For example, a control may represent three letters and a number, and three roll positions may be used to select among the three characters. Two pitch positions may select the case for the character, and a third pitch tilt position may select the number represented by the key.

[0357] Additionally, the tilt sensor independently indicates whether the device is rolled left, neutral, or right, or whether the device is pitched forward, neutral, or backward, thereby indicating whether the device is in one of nine orientations, each of which may correspond to a character and a case for the character.

[0358] FIG. 37 is a table showing one possible mapping of device orientation to output signals corresponding to letters and cases that may be output when the control marked "2" on the keypad is selected. In the illustrated mapping, a device rolled left and pitched forward causes the output of the capital letter "A." A device that is not rolled or pitched in either direction causes the output of the lowercase letter "b." And a device pitched backward causes the output of the digit "2." In other embodiments where the tilt sensor can distinguish between more than two roll positions or more than two pitch positions, more orientations in which letters and cases can be positioned are available.

[0359] An output signal corresponding to a character is described as being selected based on the angular displacement or tilt position of a first axis of the device, and an output signal corresponding to the uppercase or lowercase case of a character is described as being selected based on the angular displacement or position of a second axis of the device as a whole. In other embodiments, angular displacement of separate axes may achieve output of a signal corresponding to a character or the uppercase or lowercase case of a character. In general, any orientation of the device may be positioned for any character and case for a character, regardless of which axis is used to select the character or case.

[0360] In addition to outputting signals corresponding to letters output in response to control selection, the device orientation may be used to indicate a menu option to be selected. For example, selecting a control that does not correspond to any letter (e.g., the "1" key on a phone) causes the phone's display to present a menu (each option on the menu corresponds to a different orientation on the phone). When a control that indicates a selection must be made from a menu (e.g., the "OK" key, the "Enter" key, or the "1" key) is selected, the device orientation may indicate which of the menu options will be selected. In one embodiment, when the "1" key is selected, a menu of symbols similar to those illustrated in FIGS. 38A and 38B is displayed. Tilting the device and selecting the "1" key as before may cause the corresponding symbol to be output. After the symbol is output, letters and numbers may be output as described above until the "1" key is selected to display the symbol menu as before. Turning the device completely upside down, shaking the device, or otherwise moving the device in a manner that is not interpreted as tilting the device generates another menu.

[0361] If the angular displacement is within the first tilt region when the selection is received, a first output signal is output, whereas if the angular displacement is within the second tilt region when the selection is received, a second output signal is output, and further, if the angular displacement is within the third or fourth tilt region when the selection is received, a third or fourth output signal is output, respectively.

[0362] When a first-axis tilt region is defined about the first axis and a second-axis tilt region is defined about the second axis, one of the first plurality of output signals may be output based on the first-axis tilt region and / or the second-axis tilt region. When a selection is received, a first output signal may be output if the first-axis component is within the first first-axis tilt region and the second-axis component is within the first second-axis tilt region. A second output signal may be output if the first-axis component is within the second first-axis tilt region and the second-axis component is within the first second-axis tilt region. A third output signal may be output if the first-axis component is within the second first-axis tilt region and the second-axis component is within the second second-axis tilt region. And / or, a fourth output signal may be output when the first axis component is within a second first axis tilt region and the second axis component is within a second second axis tilt region.

[0363] Alternatively, in another aspect, when a selection is received, a first output signal may be output if the first component is in a first first-axis tilt region and the second component is in a first second-axis tilt region. A second output signal may be output if the first component is in a first first-axis tilt region and the second component is in a second second-axis tilt region. A third output signal may be output if the first component is in a first first-axis tilt region and the second component is in a third second-axis tilt region. A fourth output signal may be output if the first component is in a second first-axis tilt region and the second component is in a first second-axis tilt region. A fifth output signal may be output if the first component is in a second first-axis tilt region and the second component is in a second second-axis tilt region. A sixth output signal may be output when the first component is in a second first-axis tilt region and the second-axis component is in a third second-axis tilt region. A seventh output signal may be output when the first component is in a third first-axis tilt region and the second-axis component is in a first second-axis tilt region. An eighth output signal may be output when the first component is in a third first-axis tilt region and the second-axis component is in a second second-axis tilt region. And / or a ninth output signal may be output when the first component is in a third first-axis tilt region and the second-axis component is in a third second-axis tilt region.

[0364] The output signal is displayed (step ES310), and method E300 ends (step ES311). The output signal is displayed on a display, such as display E105. In an alternative embodiment, the output signal is not displayed.

[0365] In the embodiment of FIG. 32, device E500 also includes display E505, which is used to present a graphical user interface ("GUI") to a user of device E500. The GUI enables a user of device E500 to perform functions that require the user to enter text into device E500. For example, a user may identify an entry for a person within a phone book stored on device E500 by entering the person's name. As another example, a user may add an entry for a person to the phone book by entering information describing the person (e.g., the person's name and one or more phone numbers used by the person). Furthermore, the GUI enables a user to specify text messages to be sent from device E500 or specify other text notes to be stored on device E500. Device E500 also displays a GUI that enables a user to specify text messages.

[0366] Interpreting a control selection based on the device orientation when the control selection is made increases the number of operations that may be performed by a single control selection. For example, each control selection may be interpreted in a number of ways equal to the number of different device orientations that may be detected. Furthermore, the device orientation may indicate how a control selection that does not correspond to any character may be interpreted. Thus, a user may be able to quickly perform relatively complex operations by simply tilting the device to select a control. For example, selecting the "*" key while the device is rolled left may cause a particular mode of text input (e.g., numbers only, all uppercase) to be used for text input until the next time the "*" key is selected when the device is rolled left. In another aspect, a tilt sensor achieves tilt scrolling. Then, upon receiving a control selection, the user interface is scrolled corresponding to the direction of the tilt. For example, a forward pitch occurring when selecting a control results in the user interface or menu items on the user interface scrolling upward.

[0367] According to another general aspect, a computer program product tangibly stored on a computer-readable medium is described. The computer program product is operable to cause a computer to perform operations including determining a neutral position, about at least a first axis, of a device including a first control associated with at least a first plurality of output signals and measuring an angular displacement of the device about at least the first axis. The computer program product is also operable to cause the computer to perform operations including receiving a selection of the first control and outputting one of the first plurality of output signals based on at least the selection and the angular displacement.

[0368] Finally, although many embodiments have been described or illustrated as telephone devices, the concepts related herein are in no way limited to telephones and are in fact believed to be applicable to a wide variety of devices, including any device in which the number of controls is minimized due to device design and layout constraints. Sample devices include computer keyboards, remote controls, watches, joysticks or game controllers, or other computer input or consumer electronic devices.

[0369] Thus, a number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different embodiments may be combined, supplemented, or removed to create other embodiments. Furthermore, various technologies may be used, combined, and modified to create the embodiments. Such technologies include, for example, various digital electronic circuits, hardware, software, firmware, integrated components, discrete components, processing devices, memory storage devices, communication devices, lenses, filters, display devices, and projection devices.

[0370] (Game System) A gaming system 39 according to some embodiments will be described with reference to Figure 39. Figure 39 is an external view illustrating the gaming system 39. In the following description, the gaming system 39 according to some embodiments includes a fixed gaming device.

[0371] As shown in FIG. 39, the game system F1 includes a fixed game device (hereinafter simply referred to as the "game device") F3. This device is connected via a connection cord to a display (hereinafter referred to as the "monitor") F2, such as a home television set with a speaker F2a, and to a controller F7 for providing operational information to the game device F3. The game device F3 is connected via a connection terminal to a receiving unit F6. The receiving unit F6 receives data wirelessly transmitted from the controller F7. The controller F7 and the game device F3 are connected to each other via wireless communication. An optical disc F4, an example of an exchangeable information storage medium, is removably mounted on the game device F3. The game device F3 includes a power on / off switch, a game process reset switch, and an open switch for opening the top cover of the game device F3, on the top main surface of the game device F3. When a player presses the open switch, the cover is opened. As a result, the optical disc F4 can be mounted or removed.

[0372] Furthermore, the game device F3 is equipped with an external memory card F5 that can be detached as needed. The external memory card F5 includes a backup memory for permanently storing saved data and the like. The game device F3 executes a game program stored on an optical disc F4 and displays the results as game images on the monitor F2. The game device F3 can also use the saved data stored on the external memory card F5 to replay the state of a game played in the past and display the game images on the monitor F2. A player playing on the game device F3 can enjoy the game by operating the controller F7 while viewing the game images displayed on the monitor F2.

[0373] The controller F7 wirelessly transmits data from a communication unit F75 (described below) contained therein to the game device F3 connected to the receiving unit F6, using, for example, Bluetooth® technology. The controller F7 has two control units: a core unit F70 and a subunit F76, which are interconnected by a flexible connection cable F79. The controller F7 is primarily an operation means for operating player objects appearing in the game space displayed on the monitor F2. The core unit F70 and the subunit F76 each include an operation unit (e.g., multiple operation buttons, keys, sticks, etc.). As described in detail below, the core unit F70 includes an imaging information calculation unit F74 for capturing an image viewed from the core unit F70. As an example of an imaging target for the imaging information calculation unit F74, two LED modules F8L and F8R are provided near the display screen of the monitor F2. The LED modules F8L and F8R each output infrared light forward from the monitor F2. In this embodiment, although the core unit F70 and the subunit F76 are connected to each other by a flexible cable, the subunit F76 may have a wireless unit, thereby eliminating the need for the connecting cable F79. For example, the subunit F76 may have a Bluetooth® unit as its wireless unit, thereby enabling the subunit F76 to transmit operation data to the core unit F70.

[0374] Next, the structure of the game device F3 will be described with reference to Figure 40. Figure 40 is a functional block diagram of the game device F3.

[0375] As shown in FIG. 40, the game device F3 includes, for example, a RISC CPU (Central Processing Unit) F30 for executing various types of programs. The CPU F30 executes a boot program stored in a boot ROM (not shown) to initialize the main memory F33, and then executes a game program stored on an optical disk F4 to execute a game process according to the game program. The CPU F30 is connected to a GPU (Graphics Processing Unit) F32, main memory F33, a DSP (Digital Signal Processor) F34, and an ARAM (Audio RAM) F35 via a memory controller F31. The memory controller F31 is connected to a controller I / F (Interface) F36, a video I / F F37, an external memory I / F F38, an audio I / F F39, and a disk I / F F41 via predetermined buses. The controller I / F F36, video I / F F37, external memory I / F F38, audio I / F F39, and disk I / F F41 are connected to the receiving unit F6, monitor F2, external memory card F5, speaker F2a, and disk drive F40, respectively.

[0376] The GPU F32 performs image processing based on instructions from the CPU F30. The GPU F32 includes a semiconductor chip for performing the calculation processes necessary to display, for example, 3D images. The GPU F32 performs image processing using memory dedicated to image processing (not shown) and a portion of the storage area of ​​the main memory F33. The GPU F32 generates game image data and movies to be displayed on the monitor F2 using this type of memory, and outputs the generated data or movies to the monitor F2 via the memory controller F31 and the video I / F F37 as needed.

[0377] The main memory F33 is a storage area used by the CPU F30 and stores, as needed, game programs and the like required for processing executed by the CPU F30. For example, the main memory F33 stores game programs and the like, such as various types of data, that are read by the CPU F30 from the optical disc F4. The game programs and the various types of data stored in the main memory F33 are executed by the CPU F30.

[0378] The DSP F34 processes sound data generated by the CPU F30 while a game program is running. The DSP F34 is connected to an ARAM F35 for storing sound data. The ARAM F35 is used when the DSP F34 executes a specific process (for example, storing a game program or previously loaded sound data). The DSP F34 reads the sound data stored in the ARAM F35 and outputs the sound data to the speaker F2a included in the monitor F2 via the memory controller F31 and the audio I / F F39.

[0379] The memory controller F31 comprehensively controls data transmission and is connected to the various I / Fs described above. The controller I / F F36 includes, for example, four controller I / Fs F36a, F36b, F36c, and F36d, and communicatively connects the game apparatus F3 to external devices that can be engaged via the connectors of the controller I / Fs F36a, F36b, F36c, and F36d. For example, the receiving unit F6 engages with such a connector and is connected to the game apparatus F3 via the controller I / F F36. As described above, the receiving unit F6 receives transmission data from the controller F7 and outputs the transmission data to the CPU F30 via the controller I / F F36. The video I / F F37 is connected to the monitor F2. The external memory I / F F38 is connected to the external memory card F5 and can access backup memory and the like provided on the external memory card F5. The audio I / F F39 is connected to a speaker F2a built into the monitor F2 so that sound data read by the DSP F34 from the ARAM F35 or sound data directly output from the disk drive F40 can be output from the speaker F2a. The disk I / F F41 is connected to the disk drive F40. The disk drive F40 reads data stored at a predetermined reading position on the optical disk F4 and outputs the data to the bus of the game device F3 or the audio I / F F39.

[0380] Next, the controller F7 will be described with reference to Figures 41 and 42. Figure 41 is a perspective view illustrating the appearance of the controller F7. Figure 42 is a perspective view illustrating the state of the connection cable F79 of the controller F7 shown in Figure 41 connected to or disconnected from the core unit F70.

[0381] 41, the controller F7 includes a core unit F70 and a subunit F76 connected to each other by a connection cable F79. The core unit F70 has a housing F71 that includes multiple operation units F72. The subunit F73 has a housing F77 that includes multiple operation units F78. The core unit F70 and the subunit F76 are connected to each other by the connection cable F79.

[0382] 42, the connection cable F79 has one end thereof a connector F791 that is detachably connected to the connector F73 of the core unit F70. The other end of the connection cable F79 is fixedly connected to the subunit F76. The connector F791 of the connection cable F79 engages with the connector F73 provided on the rear surface of the core unit F70 to interconnect the core unit F70 and the subunit F76 via the connection cable F79.

[0383] The core unit F70 will be described with reference to Figures 43 and 44. Figure 43 is a perspective view of the core unit F70 seen from above and behind, and Figure 44 is a perspective view of the core unit F70 seen from below and in front.

[0384] 43 and 44, the core unit F70 includes a housing F71 formed by plastic molding or the like. The housing F71 has a generally parallelepiped shape extending longitudinally from the front to the rear. The overall size of the housing F71 is small enough to be held in one hand by an adult or even a child.

[0385] A cross key F72a is provided in the center of the front side of the top surface of the housing F71. The cross key F72a is a cross-shaped four-way push switch. The cross key F72a includes operation units corresponding to four directions (forward, backward, right, and left) represented by arrows, which are positioned on cross-shaped protrusions arranged at 90-degree intervals. The player selects one of the directions (forward, backward, right, and left) by pressing one of the operation units of the cross key F72a. By operating the cross key F72a, the player can, for example, specify the direction in which the player's character or the cursor appearing in the virtual game world moves.

[0386] Although the cross key F72a is an operation unit for outputting operation signals according to the above-described directional input operations performed by the player, this type of operation unit may be provided in other forms. For example, the cross key F72a may be replaced with a composite switch including a push switch including an annular four-direction operation unit and a central switch located in the center of the push switch. Alternatively, the cross key F72a may be replaced with an operation unit including a tiltable stick protruding from the top surface of the housing F71 and outputting operation signals according to the tilt direction of the stick. Still alternatively, the cross key F72a may be replaced with an operation unit including a horizontally slidable disk-shaped member and outputting operation signals according to the sliding direction of the disk-shaped member. Still alternatively, the cross key F72a may be replaced with a touchpad. Still alternatively, the cross key F72a may be replaced with an operation unit including switches representing at least four directions (forward, backward, right, and left) and outputting operation signals according to the switches pressed by the player.

[0387] A plurality of operation buttons F72b, F72c, F72d, F72e, F72f, and F72g are provided behind the cross key F72a on the top surface of the housing F71. The operation buttons F72b, F72c, F72d, F72e, F72f, and F72g are operation units that output respective operation signals assigned to the operation buttons F72b, F72c, F72d, F72e, F72f, and F72g when pressed by the player's head. For example, the operation buttons F72b, F72c, and F72d are assigned the functions of a first button, a second button, and an A button. Furthermore, for example, the operation buttons F72e, F72f, and F72g are assigned the functions of a minus button, a home button, and a plus button. The operation buttons F72b, F72c, F72d, F72e, F72f, and F72g are assigned respective functions according to a game program executed by the game device F3. In the exemplary device shown in Fig. 43, the operation buttons F72b, F72c, and F72d are arranged on the center line in the front-to-rear direction on the top surface of the housing F71. The operation buttons F72e, F72f, and F72g are arranged on a line in the left-to-right direction between the operation buttons F72b and F72d on the top surface of the housing F71. The top surface of the operation button F72f is recessed within the top surface of the housing F71 to prevent it from being inadvertently pressed by the player.

[0388] An operation button F72h is provided in front of the cross key F72a on the top surface of the housing F71. The operation button F72h is a power switch for remotely turning on or off the power of the game apparatus 3. The top surface of the operation button F72h is also recessed into the top surface of the housing F71 to prevent it from being accidentally pressed by the player.

[0389] A plurality of LEDs F702 are provided behind the operation buttons F72c on the top surface of the housing F71. The controller F7 is assigned a controller type (number) so that it can be distinguished from other controllers F7. For example, the LEDs F702 are used to inform the player of the controller type currently set for the controller F7 they are using. Specifically, when the core unit F70 sends transmission data to the receiving unit F6, one of the plurality of LEDs F702 corresponding to the controller type is illuminated.

[0390] On the top surface of the housing F71, sound holes for outputting sound from the speaker F706 shown in FIG. 45 to the outside are provided between the operation buttons F72e, F72f, and F72g and the operation button F72b, as will be described below.

[0391] A recessed portion is formed on the bottom surface of the housing F71. As will be described in detail later, the recessed portion is formed at a position where the player's index finger or middle finger is positioned when the player holds the core unit F70. An operation button F72i is provided on the rear inclined surface of the recessed portion. The operation button F72i is an operation unit that functions as, for example, a B button. The operation button F72i is used, for example, as a trigger switch in a shooting game or for attracting the player's attention to a predetermined object.

[0392] An image pickup element F743 included in the imaging information calculation unit F74 is provided on the front surface of the housing F71. The imaging information calculation unit F74 is a system for analyzing image data captured by the core unit F70 and detecting the centroid, size, and other characteristics of areas with high brightness in the image data. The imaging information calculation unit F74 has a maximum sampling interval of, for example, approximately 200 frames per second, and is therefore able to track and analyze even relatively fast movements of the core unit F70. The imaging information calculation unit F74 will be described in detail below. A connector F73 is provided on the rear surface of the housing F71. The connector F73 is, for example, a 32-pin edge connector, and is used to engage and connect the core unit F70 to a connector F791 of the connection cable F79.

[0393] The internal structure of the core unit F70 will be described with reference to Figures 45 and 46. Figure 45 is a perspective view illustrating a state in which the upper casing (part of the housing F71) of the core unit F70 has been removed, as viewed from the rear side of the core unit F70. Figure 46 is a perspective view illustrating a state in which the lower casing (part of the housing F71) of the core unit F70 has been removed, as viewed from the front side of the core unit F70. Figure 46 is a perspective view illustrating the back side of the board F700 shown in Figure 45.

[0394] As shown in FIG. 45, the substrate F700 is fixed inside the housing F71. Operation buttons F72a, F72b, F72c, F72d, F72e, F72f, F72g, and F72h, an acceleration sensor F701, an LED F702, an antenna F754, and the like are provided on the upper main surface of the substrate F700. These elements are connected to a microcomputer F751 (see FIGS. 46 and 55) and the like via lines (not shown) formed on the substrate F700 and the like. The core unit F70 can function as a wireless controller via a wireless module F753 (see FIG. 55) and the antenna F754 (not shown). A crystal oscillator F703 (not shown) provided in the housing F71 generates a reference clock for the microcomputer F751 (described later). A speaker F706 and an amplifier F708 are provided on the upper main surface of the substrate F700. The acceleration sensor F701 is provided near an edge offset from the center of the substrate F700. Thus, acceleration including changes in the direction of gravitational acceleration and centrifugal force components can be detected based on the rotation of the core unit F70 about its longitudinal direction. As a result, a predetermined calculation can be used to determine the rotation of the core unit F70 with a desired accuracy based on the detected acceleration data.

[0395] As shown in FIG. 46, an imaging information calculation unit F74 is provided on the front edge of the bottom main surface of the substrate F700. The imaging information calculation unit F74 includes an infrared filter F741, a lens F742, an image pickup element F743, and an image processing circuit F744, located in this order from the front face of the core unit F70 on the bottom main surface of the substrate F700. A connector F73 is attached to the rear edge of the bottom main surface of the substrate F700. Furthermore, a sound IC F707 and a microcomputer F751 are provided on the bottom main surface of the substrate F700. The sound IC F707, which is connected to the microcomputer F751 and the amplifier F708 via lines formed on the substrate F700, outputs an audio signal to the speaker F706 via the amplifier F708 based on sound data transmitted from the game device F3. A vibrator F704 is provided on the bottom main surface of the substrate F700. The vibrator F704 is, for example, a vibration motor or a solenoid. The core unit F70 vibrates when the vibrator F704 is activated. The vibration is then transmitted to the hand of the player holding the core unit F70. Thus, a so-called vibration feedback game is realized. The vibrator F704 is disposed slightly toward the front of the housing F71. This allows the housing F71 held by the player to vibrate strongly, allowing the player to easily sense the vibration.

[0396] The subunit F76 will be described with reference to Figures 47 to 50. Figure 47 is a perspective view illustrating a first embodiment of the subunit F76. Figure 48 is a perspective view illustrating the subunit F76 shown in Figure 47 with the upper casing (part of the housing F77) removed. Figure 49A is a top view illustrating a second embodiment of the subunit F76. Figure 49B is a bottom view illustrating the second embodiment of the subunit F76. Figure 49C is a left side view illustrating the second embodiment of the subunit F76. Figure 50 is a perspective view illustrating the second embodiment of the subunit F76 as seen from the upper front side.

[0397] As shown in Fig. 47, the subunit F76 includes a housing F77 formed, for example, by plastic molding. The housing F77 extends longitudinally from front to rear and has a streamlined three-dimensional shape including a head, which is the widest part of the subunit F76. The overall size of the subunit F76 is small enough to be held in one hand by an adult or even a child.

[0398] A stick F78a is provided near the widest part of the top surface of the housing F77. The stick F78a includes a tiltable stick protruding from the top surface of the housing F77 and is an operation unit that outputs an operation signal according to the tilt direction of the stick. For example, a player can arbitrarily indicate a direction and position by tilting the tip of the stick in any direction within 360 degrees. This allows the player to indicate the direction in which the player's character or other characters appearing in the virtual game world will move, or the direction in which the cursor will move. 【03...

Claims

1. A method for managing a game system having a light source at a known location, the method is The steps include: the light source emitting light in the direction of a first sensor attached to the limb of one or more of the players, wherein the light source identifies the movement of the first sensor relative to the light source; A step of generating a first signal that represents multiple game commands and corresponds to the multidimensional movement of the first sensor, The steps include receiving and converting the first signal in order to control a game of chance that is played on a game device and to provide input to this game, The process includes providing feedback to the first sensor in order to notify at least one of the multiple players of the outcome of the game, which is determined by chance, method.

2. The method according to claim 1, wherein the game device receives the first signal from one or more of the players among the plurality of players, the first signal is for identifying the usage status of one or more of the players among the plurality of players and the game device, and after receiving the first signal, the game device requests identification information (ID) of one or more of the players among the plurality of players.

3. The method according to claim 1, further comprising the first sensor detecting at least one of the pulse rate, body temperature, skin conductivity, skin moisture content, skin electric field, muscle tone, and other biosignals of one or more of the players.

4. The method according to claim 1, further comprising the step of communicating the ID of one or more of the aforementioned players between the first sensor and the game device to start a game session.

5. The method according to claim 4, further comprising the step of requesting one or more of the aforementioned players to re-identify and / or verify their IDs.

6. The method according to claim 1, wherein the first sensor includes a plurality of operating modes.

7. A device, and said device is Motion sensor and, Electromagnetic transceiver and, A haptic transducer for controlling a game system having a light source at a known location, Includes at least one processor, The at least one processor is The light source emits light in the direction of a first sensor attached to the limb of one or more of the players, and the movement of the first sensor relative to the light source is identified by the light source. To generate a first signal that represents multiple game commands and corresponds to the multidimensional movement of the first sensor, Controlling a game of chance played on a game device, receiving and converting the first signal in order to provide input to this game, and The system is configured to control the operation of providing feedback to the first sensor in order to notify at least one of the multiple players of the outcome of the game which is determined by luck, device.

8. The device according to claim 7, wherein the game device receives the first signal from one or more of the players among the plurality of players, the first signal is for identifying the usage status of one or more of the players among the plurality of players and the game device, and after receiving the first signal, the game device requests identification information (ID) of one or more of the players among the plurality of players.

9. The apparatus according to claim 8, wherein the at least one processor is further configured so that the first sensor can detect at least one of the pulse rate, body temperature, skin conductivity, skin moisture content, skin electric field, muscle tone, and other biosignals of one or more of the players.

10. The apparatus according to claim 8, wherein the at least one processor is further configured to communicate the ID of one or more of the players among the plurality of players between the first sensor and the game device to start a game session.

Images