Games with hand motion controls

Abstract

For games with hand motion controls. The method includes receiving a first wireless signal from a first device, receiving a second wireless signal from a second device, determining a first player identifier from the first wireless signal, determining a second player identifier from the second wireless signal, displaying a message requesting the player to identify themselves, receiving an indication of a third player identifier via a tactile input, determining that the third player identifier matches the first player identifier, receiving a third wireless signal from the first device, interpreting the third wireless signal as a command in a gambling game, and executing the command in the gambling game.

Description

Technical Field

[0001] This application claims the benefit of priority of U.S. Patent Application No. 11 / 754,944, filed May 29, 2007, entitled "Game With Hand Motion Control". The entire disclosure of the above application is incorporated herein by reference. This application relates to games having hand motion control.

Background Art

[0002] This application relates to games having hand motion control.

Summary of the Invention

Means for Solving the Problems

[0003] The present invention includes receiving a first wireless signal from a first device, receiving a second wireless signal from a second device, determining a first player identifier from the first wireless signal, determining a second player identifier from the second wireless signal, displaying a message requesting the player to identify himself, receiving an indication of an identifier of a third player via a tactile input portion, determining that the identifier of the third player matches the identifier of the first player, receiving a third wireless signal from the first device, interpreting the third wireless signal as a command in a gambling game, and executing the command in the gambling game. A method is provided.

Brief Description of the Drawings

[0004] [Figure 1] FIG. 1 shows a game system according to some embodiments. [Figure 2] FIG. 2 shows a communication network according to some embodiments. [Figure 3]Figure 3 shows a game service provider communicating with a game communication device according to one embodiment. [Figure 4] Figure 4 shows a game network according to one embodiment. [Figure 5] Figure 5 shows a game system according to one embodiment. [Figure 6] Figure 6 shows a wireless game system according to one embodiment. [Figure 7] Figure 7 shows a portable game device having promotional content according to one embodiment. [Figure 8] Figure 8 is a block diagram of a game system according to one embodiment. [Figure 9] Figure 9 is a block diagram of a payment system that forms part of the game system shown in Figure 8, according to one embodiment. [Figure 10] Figure 10 is a schematic diagram of a portable game device of the game system shown in Figure 8, according to one embodiment. [Figure 11a] Figure 11(a) is a flowchart illustrating how a player uses a portable game device according to one embodiment. [Figure 11b] Figure 11(b) is a flowchart illustrating a specific method of using a portable game device by a player, according to one embodiment. [Figure 12] Figure 12 is a flowchart illustrating how a game service operator uses a portable game device, according to one embodiment. [Figure 13] Figure 13 is a flowchart illustrating a method of using a portable game device according to one embodiment. [Figure 14a] Several one-camera-based embodiments are shown. [Figure 14b] Several embodiments of 3-D (three-dimensional) detection are shown. [Figure 14c] Several embodiments are shown that feature two cameras, a "binocular" stereo camera. [Figure 14d] Several steps following some embodiments are shown. [Figure 14e]This document describes a process for color mapping according to several embodiments. [Figure 15] The hardware components and physical layout of a multi-camera control system implementation according to several embodiments are shown. [Figure 16A] The geometric relationships between the camera and various image regions in Figure 15 are shown according to several embodiments. [Figure 16B] Figure 15 shows an image captured by one of the cameras according to several embodiments. [Figure 17] This flowchart illustrates a process executed within a microcomputer program associated with a multi-camera control system, according to several embodiments. [Figure 18] This flowchart shows in more detail a portion of the process shown in Figure 17, according to several embodiments, particularly the process related to detecting an object from an image signal captured by a camera and extracting its position. [Figure 19A] Sample image data presented as a grayscale bitmap image is shown, acquired by a camera according to several embodiments and generated by part of the process shown in Figure 18. [Figure 19B] Sample image data presented as a grayscale bitmap image generated by part of the process shown in Figure 18, according to several embodiments, is shown. [Figure 19C] Sample image data presented as a grayscale bitmap image generated by part of the process shown in Figure 18, according to several embodiments, is shown. [Figure 19D] Sample image data presented as a grayscale bitmap image generated by part of the process shown in Figure 18, according to several embodiments, is shown. [Figure 19E]Sample data presented as a binary bitmap image identifying pixels in a sample that are likely to belong to a tracked object, generated by a part of the process shown in FIG. 18, according to some embodiments. [Figure 20] A flowchart that more particularly shows a process related to classifying and identifying an object when, according to some embodiments, a map of pixels identified as likely to belong to a tracked object, such as the data shown in FIG. 19E, is provided as part of the process described in FIG. 18. [Figure 21A] Sample data presented as a binary bitmap image, presented in FIG. 19E, showing the identification of data samples selected as belonging to an object in this sample, along with the process shown in FIG. 20, according to some embodiments. [Figure 21B] Sample data presented in FIG. 19E as a bar graph, showing the identification of data samples selected as belonging to an object, along with the process outlined in FIG. 20, according to some embodiments. Specific points in the graph are identified. [Figure 21C] A different set of sample data presented as a binary bitmap image, showing the identification of data samples selected as belonging to an object and the key part of that object in this sample, along with the process shown in FIG. 20, according to some embodiments. [Figure 22] A flowchart that more particularly shows a process related to generating and maintaining a description of the background region blocked by an object, as part of the process shown in FIG. 18, according to some embodiments. [Figure 23A] According to some embodiments, Equation 3 underlies an arrangement that defines an angle that positions an object within the camera's field of view when the position of the object on the image plane where it was detected is given. [Figure 23B] According to some embodiments, Equations 4, 5, and 6 underlie an arrangement that shows the relationship between the position of the camera and a tracked object. [Figure 24] A graph according to some embodiments, Equation 8, that is, for refining a position, showing an attenuation amount that may be added to coordinates when a change in the position of an object is given. [Figure 25A] An example of an application program controlled by a system in which an object of interest controls a screen pointer two-dimensionally, according to some embodiments. [Figure 25B] Shows the mapping between real-world coordinates and screen coordinates used by the application program of FIG. 25A, according to some embodiments. [Figure 26A] An example of an application program controlled by a multi-camera control system in which an object of interest controls a screen pointer within a three-dimensional virtual reality environment, according to some embodiments. [Figure 26B] An example of an application program controlled by a multi-camera control system in which an object of interest controls a screen pointer within a three-dimensional virtual reality environment, according to some embodiments. [Figure 27A] Shows the division of regions into detection surfaces used by a gesture detection method for identifying gestures that may be associated with an intention to activate, according to some embodiments. [Figure 27B] Shows the division of a region of interest into a detection box used by a gesture detection method for identifying gestures that may be associated with selecting a cursor direction, according to some embodiments. [Figure 27C] Shows an alternative division of regions into a direction detection box used by gesture detection for identifying gestures that may be associated with selecting a cursor direction, according to some embodiments. [Figure 27D] Shows in more detail the relationship between adjacent sections of FIG. 27C, according to some embodiments. [Figure 28] Shows the appearance of a device according to some embodiments in a state where the device is in a neutral position. [Figure 29] An example of the internal structure of the implementation shown in Figure 28, according to several embodiments, is presented. [Figure 30] This flowchart illustrates a method relating to another exemplary implementation, according to several embodiments. [Figure 31A] Figure 31A shows an example of a sloped region defined around the neutral axis, according to several embodiments. [Figure 31B] Figure 31B shows an example of a sloped region defined around the neutral axis, according to several embodiments. [Figure 31C] Figure 31C shows an example of a sloped region defined around the neutral axis, according to several embodiments. [Figure 31D] Figure 31D shows an example of a sloped region defined around the neutral axis, according to several embodiments. [Figure 32] An external top view of an exemplary device relating to another exemplary implementation according to several embodiments is shown. [Figure 33A] Figure 33A shows an exemplary indicator according to several embodiments. [Figure 33B] Figure 33B shows an exemplary indicator according to several embodiments. [Figure 33C] Figure 33C shows an exemplary indicator according to several embodiments. [Figure 33D] Figure 33D shows an exemplary indicator according to several embodiments. [Figure 33E] Figure 33E shows an exemplary indicator according to several embodiments. [Figure 34A] Figure 34A shows a front view and a side view of the apparatus of Figure 32, respectively, shown in the neutral position, according to several embodiments. [Figure 34B] Figure 34B shows a front view and a side view of the apparatus of Figure 32, respectively, shown in the neutral position, according to several embodiments. [Figure 35A]Figure 35A shows a front view of the apparatus of Figure 32 in which it is operated in negative roll orientation and positive roll orientation, respectively, according to several embodiments. [Figure 35B] Figure 35B shows a front view of the apparatus of Figure 32 in which it is operated in negative roll orientation and positive roll orientation, respectively, according to several embodiments. [Figure 36A] Figure 36A shows a side view of the apparatus of Figure 32 in which it is operated in positive pitch orientation and negative pitch orientation, respectively, according to several embodiments. [Figure 36B] Figure 36B shows a side view of the apparatus of Figure 32 in which it is operated in positive pitch orientation and negative pitch orientation, respectively, according to several embodiments. [Figure 37] This is a table showing one possible mapping of device orientation used to output signals corresponding to characters and cases, which are outputs when a control is selected, according to several embodiments. [Figure 38A] Figure 38A shows a menu of symbols as displayed according to another exemplary implementation, which follows several embodiments. [Figure 38B] Figure 38B shows a menu of symbols as displayed according to another exemplary implementation, which follows several embodiments. [Figure 39] This is an external view showing a game system F1 according to several embodiments. [Figure 40] Figure 39 is a functional block diagram of the game device F3. [Figure 41] Figure 39 is a perspective view showing the external appearance of controller F7. [Figure 42] This is a perspective view showing the state of the connection cable F79 of the controller F7 shown in Figure 41, which is being connected to or disconnected from the core unit F70. [Figure 43] Figure 41 is a perspective view of the core unit F70, as seen from above and behind. [Figure 44] Figure 41 is a perspective view of the core unit F70, as seen from the front of the bottom surface. [Figure 45] Figure 41 is a perspective view showing the upper casing of the core unit F70 with it removed. [Figure 46] Figure 41 is a perspective view showing the core unit F70 with its lower casing removed. [Figure 47] Figure 41 is a perspective view showing a first example of subunit F76. [Figure 48] Figure 47 is a perspective view showing the upper casing of subunit F76 with it removed. [Figure 49A] Figure 49A is a top view of a second example of the subunit F76 shown in Figure 41. [Figure 49B] Figure 49B is a bottom view of a second example of subunit F76 shown in Figure 41. [Figure 49C] Figure 49C is a left side view of a second example of the subunit F76 shown in Figure 41. [Figure 50] Figure 41 is a perspective view of subunit F76, as seen from above and the front. [Figure 51] Figure 41 is a top view showing an example of the first deformation of subunit F76. [Figure 52] Figure 41 is a top view showing an example of a second deformation of subunit F76. [Figure 53] Figure 41 is a top view showing an example of a third deformation of subunit F76. [Figure 54] Figure 41 is a top view showing an example of a fourth deformation of subunit F76. [Figure 55] Figure 41 is a block diagram showing the structure of controller F7. [Figure 56] Figure 41 is a schematic diagram showing the state of a game being controlled using controller F7. [Figure 57]This image shows an exemplary view of the core unit F70 from the front, with the player holding the core unit F70 in their right hand. [Figure 58] This image shows an exemplary state of a player holding the core unit F70 with their right hand, viewed from the left side of the core unit F70. [Figure 59] This is a schematic diagram showing the viewing angles of LED module F8L, LED module F8R, and image pickup element F743. [Figure 60] This image shows an exemplary state of a player holding subunit F76 with their left hand, viewed from the right side of subunit F76. [Figure 61] This shows an exemplary game image displayed on monitor F2 when game device F3 is running a shooting game. [Modes for carrying out the invention]

[0005] In various embodiments, a player may use motion as input to a game played on a portable gaming device. This game may be a gambling game such as a raw data game, a slot machine game, a roulette game, a craps game, or any other gambling game. A player may place bets in the game and may be in a position to win money depending on the outcome of the game. A player may also be in a position to lose money in the game.

[0006] The motion used as input may include the motion of the portable game device itself. Therefore, the player may tilt, shake, move, rotate, or otherwise move the portable game device. Such movement of the portable game device may be interpreted by hardware sensors and / or software as commands or instructions for playing the game. Thus, one motion may be seen as a signal to start the game or a signal for cash payout.

[0007] In various embodiments, the player may be provided with audio feedback. This audio feedback may be given in accordance with motions performed by the player or in accordance with motions recognized by the portable game device. The audio feedback may be given during motions performed by the player. This audio feedback may enhance the player's gaming experience by providing the player with sounds similar to those one might hear while playing a game on a standalone game machine such as an actual game cabinet or slot machine. The audio feedback may provide information to the player. The audio feedback may inform the player that a motion performed by the player was recognized as a command, or that a motion performed by the player was not recognized as a command.

[0008] In various embodiments, the player may be provided with force feedback or tactile feedback. The portable game device may create tactile sensations using springs, motors, resistors, or other devices that can produce motion, pressure, heat, or other tactile sensations or other sensations. The tactile feedback may give the player the sensation of shaking the portable game device in their hand, for example, as if they were rolling dice.

[0009] In various embodiments, the player may have a wristband. This wristband may include motion sensors, such as an accelerometer, for detecting motion. The player may move the hand wearing the wristband in a specific way to issue commands into the game. In various embodiments, the wristband may provide tactile feedback.

[0010] (Wristband / Bracelet) In various embodiments, the player may wear a bracelet, watch, wristband or other device around their wrist. This wristband may include one or more of the following: (a) a processor (e.g., a semiconductor processor); (b) a power supply (e.g., a battery); (c) a motion sensor (e.g., an accelerometer; e.g., a gyroscope; e.g., a camera for measuring motion based on a changing visual image); (d) a transmitter (e.g., an antenna); (e.g., a receiver (e.g., an antenna); (f) memory (e.g., semiconductor memory); (g) a display device (e.g., a liquid crystal display screen); (h) a speaker (e.g., for transmitting audio output); (i) a tactile output device.

[0011] (The wristband records motion.) In various embodiments, the wristband may track motions performed by the player wearing the wristband. For example, motion sensors within the wristband may detect acceleration, changes in position, changes in orientation, angular displacement, paths, trajectories, or any other components of motion. The wristband may track the motion of the hand or wrist on which it is worn. The wristband may store data representing the motion. Such data may be stored, for example, in the memory of the wristband. The wristband may transmit instructions for the motions performed to another device, such as a portable gaming device, a stationary gaming device, or a casino server.

[0012] In various embodiments, the wristband may store or transmit raw data, such as data indicating any readings received from a motion sensor. In various embodiments, the wristband may translate this raw data into more condensed, or higher-level, data. For example, a series of readings from a motion sensor in the bracelet may be translated into a command. That is, a player wearing the wristband may perform a motion to give a command. The wristband may then store the command rather than the exact position of the wristband as a function of time. The wristband may transmit the command to another device, for example, via a transmitter on the wristband.

[0013] (Motion constitutes commands in a game.) In various embodiments, the motion of the wristband may be interpreted as a command in a game. For example, a player may move their hand up and down to start the reels spinning in a slot machine game. The player may also move their hand in a way that represents a command to (a) cash out; (b) hold cards in video poker; (c) discard cards in video poker; (d) double down in blackjack; (e) select one of several options in a bonus round; (f) place a bet of a certain size; (g) view a list of game commands; (h) start a bonus round; or (i) select a payline to play; or to perform any other command in a game. The wristband may store a table that associates specific motions with specific game commands. Upon receiving a sensor reading indicating a specific motion, the wristband may look up the motion corresponding to that command in its table. The wristband may then transmit the command to a portable game device, a stationary game device, or another device, such as a casino server. The casino server may relay the command to another device, such as a fixed gaming machine or a portable gaming machine. In various embodiments, this command may then be executed or followed within the game.

[0014] (The wristband communicates with the portable gaming device.) In various embodiments, the wristband may communicate with a portable gaming device. The wristband may have an antenna and a receiver for this purpose. The portable gaming device may similarly have an antenna and a receiver for communicating with other devices. The portable gaming device and the wristband may communicate via various protocols such as Bluetooth, Wi-Fi, or any other protocol.

[0015] (The wristband controls other devices.) The wristband may communicate with a portable game device, a stationary game device, or any other device. The wristband may detect player motion, such as the motion of the player's hands. The wristband may interpret this motion as a command for the device with which it is communicating. The wristband may send a command to that device, which then follows the command. In some embodiments, the wristband captures raw data, such as a series of positions of the player's wrist as a function of time. This raw data is sent to another device, which then interprets this raw data as a command.

[0016] (Communicating with multiple devices simultaneously) In various embodiments, the wristband may communicate with two or more devices. The wristband may communicate with two or more devices simultaneously. The wristband may transmit a single signal that may be received by both the first and second devices. For example, a command transmitted by the wristband may be received by both the first and second slot machines. In some embodiments, the first and second devices may transmit signals approximately simultaneously. The wristband may receive both signals.

[0017] In some embodiments, a player may have two or more devices, such as two or more fixed game machines, identify themselves. The player may provide several proof of identity, such as a player tracking card, biometric authentication, or a device (such as a wristband), with an identifier (e.g., a unique identifier) ​​that can be associated with the player. The player may permit or allow communication between the player's wristband and these two or more devices. As part of the permit, the player may agree to play the game on each of these two or more devices. Thus, in some embodiments, the player may permit two or more devices to interpret signals coming from the player's wristband as command signals to be used in the game. In some embodiments, the player may present their wristband to two or more devices. For example, the player may bring their wristband within a few inches of an RFID reader on a slot machine. This slot machine may receive signals from the RFID tag on the wristband. The device may then recognize the command received from the presented wristband, but may not recognize commands received from other wristbands. Thus, the device may accept commands from the wristband for some time. In various embodiments, commands may be accepted until several stop commands are received, until no further commands are detected (for example, if the wristband is switched off or taken out of the device's communication range), until a certain amount of time has elapsed, or until some other stop condition occurs. To resume providing motion-based commands to the device, the player may once again allow receiving and using commands from the player's wristband. For example, the player may present their wristband.

[0018] In various embodiments, a player may play on two or more game devices simultaneously. The player may perform a motion, and instructions for such a motion (e.g., commands derived from such a motion) may be sent to two or more game devices. Each of these two or more game devices may execute the command. Thus, in some embodiments, a player can easily play two or more games simultaneously while avoiding the repetition of commands for each individual game. For example, a player may use a single flick of the wrist to start a game on each of two slot machines.

[0019] In some embodiments, a first device may receive data (e.g., motion data) from a wristband. This first device may interpret this data as a command and play a game based on this command. A second device may receive the same data from the wristband. This second device may transmit the data (or its interpretation) to the player's friends or other groups so that they can follow what the player is doing. The second device may transmit game result instructions, payouts, and other occurrences related to the game played by the player to the player's friends or other groups. In some embodiments, a player may play several games simultaneously using motion from their wristband. Data from those games (e.g., results) may be transmitted to a casino server or another device. The data may be made available for viewing by other groups, for example, the player's friends, or by others who will play their own games using random occurrences that occur in the player's games (for example, others may bet on the results that occur in the player's games).

[0020] In various embodiments, a player may play on two game devices simultaneously. However, each command produced by the player (e.g., through motion) may apply to only one game device at a time. For example, a player may produce a first command that applies only to the first game on the first game device. The player may then produce a second command that applies only to the second game on the second game device. The player may then produce a third command that applies only to the first game on the first game device. In various embodiments, the two game devices may be controllable by their own sets of motion commands, with little to no overlap between the motions used in the commands. Thus, for example, a motion produced by a player may correspond to a valid command on one of the game devices but not on the other. Different motions may not correspond to a valid command on the first game device but may correspond to a valid command on the second game device.

[0021] (Time when no data stream is received from the wristband) In various embodiments, the device may be within the communication range of a wristband transmitting data, but may not receive the data, interpret the data, or use the data. The device may be a portable game device or a stationary game device such as a slot machine. The device may not use data from the wristband if one or more of the following are true: (a) the player wearing the wristband has not identified themselves to the device; (b) the player wearing the wristband has not presented identification to the device; (c) the wristband is sending a command that the device does not understand; (d) the player wearing the wristband has not made at least some physical contact with the device (e.g., pressing a button on the device); (e) the player has not informed the device that the device should be anticipating motion commands from the wristband; (f) the device is currently accepting motion commands from a different wristband; (g) the player does not have a sufficiently high credit balance to play the game on the device (e.g., the player has a credit balance of zero); (h) the player has not made physical contact with the device for a given period of time (e.g., the player has not physically pressed a button on the game device in the last 10 minutes); or any other circumstances apply.

[0022] (Biometric authentication as game input) In various embodiments, the wristband may sense pulse, temperature, skin conductivity, moisture level, electric field (e.g., from nerve pulses), muscle tension, or any other biometric signals from the player. These signals may be translated into numbers. For example, a numerical temperature reading in Fahrenheit may be used as a seed for a random number generator, which is then used to generate results in a game.

[0023] In various embodiments, a biometric reading received by a wristband may indicate that the wristband is still being worn. If the wristband detects a pulse, for example, the wristband or another device may infer that the wristband is being worn by a player and has not yet been removed. In various embodiments, a portable gaming device, a stationary gaming device, or another device may act based on signals received from a wristband only if the wristband is currently being worn (or appears to be being worn based on biometric signals received from the wristband). In some embodiments, if there is an interruption in the biometric signal received by a wristband (for example, the wristband no longer detects a pulse), the wristband may send a signal to a casino server or some other device. This signal may indicate that there has been an interruption in the biometric signal detected by the wristband. Accordingly, the casino server may instruct other devices not to follow any commands or signals received from the wristband until the wristband is re-established with the player. In some embodiments, the wristband must be re-established on the player in the presence of a casino representative, or with the help of a casino representative, before the signals from the wristband are honored by another device. In some embodiments, if there is an interruption in the biometric signal detected by the wristband, the wristband may transmit a signal to call medical personnel. For example, the wristband may transmit a signal to the casino server indicating that a pulse is no longer being detected.

[0024] (The wristband broadcasts data that identifies the user.) In various embodiments, the wristband may transmit or broadcast data that identifies the player wearing the wristband. The wristband may broadcast a card number that tracks the player, the player's name, the player's alias, the player's room number, the player's credit card number, or any other information about the player that may be used to identify the player. In some embodiments, the wristband may transmit a signal derived from a biometric reading. For example, the wristband may broadcast a signal derived from a pulse or electrocardiogram reading obtained from the player. This biometric reading may serve to uniquely identify the player.

[0025] In various embodiments, signals broadcast from a wristband and identifying a player may grant certain privileges to the player wearing the wristband. The player's hotel room door may be unlocked remotely (for example, the door may be unlocked with a key or other device without requiring physical contact). The hotel room door may be unlocked when it receives a signal from the player's wristband identifying the player. The player may be permitted to gamble on certain gaming machines. The player may be permitted to enter certain areas of the casino based on the identity provided by the player's wristband. In various embodiments, the wristband may provide the player with an identifier that allows the player to access their funds or another financial account. The player may use those funds, for example, to gamble or to shop. For example, the player may approach a gaming machine. The player may have an account with a positive balance stored on a casino server. When the player's wristband transmits a player identifier to a slot machine, the slot machine may receive the identifier and transmit instructions for that identifier to the casino server. Next, the casino server may allow the player to access the player's funds. Then some or all of the player's funds may be made available for use on the gaming equipment (for example, in the form of credit balance). The player may then use those funds to play games.

[0026] In various embodiments, the wristband may be power-constrained due to the small available volume within the wristband containing a battery or other power source. The wristband may take various steps to conserve power. In some embodiments, the wristband may periodically transmit signals to another device, such as a portable game device or a stationary game device. For example, the wristband may transmit a signal consisting of a series of bits to the portable game device every 50 milliseconds. This signal may contain data or information describing the motion produced by the wristband since the last signal transmission. In various embodiments, the time between signal transmissions may vary depending on what data or information needs to be transmitted by the wristband. For example, if the wristband remains motionless, the time between signal transmissions may be extended to 200 milliseconds. When the wristband starts moving again, the time between signal transmissions may be shortened again to 50 milliseconds. Thus, in various embodiments, the time between when a signal is transmitted by the wristband may vary based on the motion of the wristband and / or the motion detected by the wristband. In various embodiments, the time intervals between times when a signal is transmitted by the wristband may vary based on the amount of information the wristband needs to communicate to another device. For example, if a player is actively involved in the game, the wristband may transmit signals frequently. If a player is not actively involved in the game (e.g., the player has not started playing the game on a stationary or portable game device; e.g., the player is not in an area where playing the game is permitted), the wristband may transmit signals relatively infrequently. In various embodiments, if the wristband is not moving, the wristband may periodically transmit short, simple signals indicating that the wristband is still operational or still usable. However, these signals may indicate that the wristband is not currently in use or is not being used for a game.

[0027] In various embodiments, the wristband may draw power or energy from the wearer's arm motion or from other motions of the wearer. The wristband may also draw energy from its own motion, and the wristband's own motion may be caused by the motion of the arm to which the wristband is attached. Examples of devices for utilizing electrical energy from motion include piezoelectric devices or mechanical rotary magnetic generators. Power sources such as those used in Fossil kinetic watches or Ventura kinetic watches may also be used.

[0028] In various embodiments, a wristband may detect relative motion between itself and another device. For example, a player may wear two wristbands. One wristband may transmit a signal of a certain intensity to the other wristband. Based on the distance between the wristbands, the signal appears relatively strong (e.g., when the wristbands are close together) or relatively weak (e.g., when the wristbands are far apart) in the receiving wristband. In this way, the proximity of the wristbands to each other may be measured. The relative motion of the wristbands may be measured to any suitable device. The player may wear the device in another location on their body, such as a belt buckle, which can transmit or receive signals. The wristbands may transmit signals to and receive signals from receivers attached to any fixed device outside the individual, such as a wall, ceiling, floor, or game device.

[0029] In various embodiments, the wristband may detect drinking motion. The wristband may detect wrist rotation via an orientation sensor within the wristband. If there is a large wrist rotation, it may be inferred that the player has almost finished drinking and therefore needs to tilt the drink further. Accordingly, the casino representative may be instructed to serve the player a new drink, and / or the player may be asked if they would like another drink.

[0030] (Technology for collecting energy for wristbands) Various techniques for collecting energy from the environment or ambient conditions are described in the paper, "Energy Scavenging for Mobile and Wireless Electronics," by Joseph A. Paradiso and Thad Starner. As of May 11, 2007, this paper was available at http: / / www.media.mit.edu / resenv / pubs / papers / 2005-02-E-HarvestingPervasivePprnt.pdf.

[0031] A radio frequency identification system allows a tag to receive energy from a remote or non-adjacent source (e.g., a tag reader). The tag receives radio frequency energy inductively, capacitively, or through heat dissipation from the tag reader.

[0032] Solar cells could be used to enable portable devices such as wristbands to draw energy from ambient light. An example of such a technology is crystalline silicon solar cells.

[0033] Thermoelectric generators may enable the induction of energy from heat transfer. These generators utilize temperature gradients, such as the difference between a person's body temperature and the ambient air temperature. Seiko's Thermic watch uses a thermoelectric generator to power its mechanical watch components. One thermoelectric technology is Thermo Life from Applied Digital Solutions.

[0034] Various technologies make it possible to collect energy from vibration or motion. Motion may be used to move a mass in a favorable or unfavorable direction. The movement of the mass may wind a spring. The energy of the spring may then be used to generate direct mechanical energy (e.g., to move the hands of a watch) or to move magnets, coils, or other components of a power generator to produce electricity. Exemplary technologies for collecting energy from mechanical motion include ETA Autoquartz, Seiko AGS (Automatic Power Generation System), and Ferro Solutions' Harvester. Piezoelectric materials may deform where there is motion or vibration to generate electricity. For example, Ocean Power Technologies has developed a harvester that is immersed in turbulent water and deforms due to the water flow to generate electricity. Some power generators include capacitors with movable plates. On a charged capacitor, the induced motion of one of the plates can generate an electric current. Piezoelectric and capacitive generators may be used, for example, to collect energy from shoes while walking.

[0035] Some power generation devices are equipped with turbines that can be driven by ambient airflow.

[0036] (Game device as an antenna array) In various embodiments, each of two or more stationary game devices may include components of an antenna array. When operating together, these game devices may detect and interpret signals from a portable game device or a wristband. For example, each of two or more stationary game devices may have an antenna. Each game device may receive signals transmitted by a portable game device or a wristband. The signals received by each of the antennas of the two or more game devices may then be amplified, possibly with some temporal delay or phase shift added by one or more game devices. Amplifying the signals received by the two or more antennas may reduce the signal-to-noise ratio, which may allow signals from the portable game device or wristband to be read with greater accuracy or over a greater distance, or may allow the portable game device to transmit with less power, thus benefiting from extended battery life.

[0037] (A new battery for each last shift) In various embodiments, the battery or power source within the wristband may be replaced regularly as is typical. The battery may be replaced (a) once a day (e.g., at the end of the day); (b) once per shift (e.g., at the end of a casino participant's shift; e.g., at the beginning of a casino participant's shift); (c) once an hour; or according to any other standard. In various embodiments, the wristband may include an indicator light or some other output device to indicate a low power level of its battery or power source. The battery may be replaced or recharged when the indicator light illuminates.

[0038] (The wristband provides the player with location information.) In various embodiments, the wristband may broadcast a signal. This signal may include a player identifier, such as a name or a card number that tracks the player. This signal may also include information about the player's location. For example, the wristband may collect location information from a beacon or satellite, calculate its own location, and transmit the location information to a game device or any receiver.

[0039] In some embodiments, the wristband measures a change in its position, but not an absolute position. A receiver receiving a signal from the wristband may be able to measure the direction of the wristband from the receiver, but not necessarily the distance to the wristband. The player wearing the wristband may then walk some distance, and the position of the wristband may change as a result. The wristband may include an accelerometer or other motion detector that can be used to measure the change in position (not necessarily an absolute position). The wristband may also include a sensor for measuring orientation, such as a compass. Thus, the wristband may measure the change in position (e.g., measured in feet or meters) and broadcast this change to the receiver. The wristband may further measure the direction in which the change in position occurred and broadcast this direction to the receiver. Again, the receiver may be able to measure the direction of the wristband from the receiver at the new position of the wristband, but not necessarily its distance from the receiver. Based on these two measurements of the direction of the wristband from the receiver, and based on the distance the wristband has moved, and based on the direction the wristband has moved, the absolute position of the wristband may be determined. This is because, in the triangle formed by the receiver, the initial and final positions of the wristband, one side, and two adjacent angles are known. This side is the path the wristband traveled (assuming it took the shortest path), and these angles can be determined based on the direction in which the receiver detected the wristband at its initial and final positions, and the way the wristband itself moved.

[0040] (A wristband used to control portable gaming devices) In various embodiments, the wristband may be used to control a portable gaming device. The wristband may transmit signals to the portable gaming device, which provide instructions or commands regarding how to proceed in the game. Such instructions may include instructions to start playing the game, instructions to hold a particular card, instructions to hit or stand (for example in blackjack), instructions to bet on a particular payline, or any other instructions. The wristband may also transmit signals to a stationary gaming device, which provide instructions or commands regarding how to proceed in the game.

[0041] The wristband may measure its own motion via a motion sensor (e.g., via an accelerometer). The wristband may interpret such motion as commands to be used in the game. To control a portable or stationary game device, the wristband may transmit such commands to such a device. In some embodiments, the wristband may record motion data such as distance moved, acceleration, trajectory, velocity, or any other motion data. The motion data may be transmitted to a portable or stationary game device. In the portable or stationary game device, the motion may be translated into game commands. In various embodiments, the wristband may transmit either motion data or game commands to a casino server. The casino server may then transmit the motion data or game commands to the portable or stationary game device to control it.

[0042] In various embodiments, the wristband may be used to control any device or to issue commands to any device. Such devices may include point-of-sale terminals, vending machines, kiosks, automated teller machines (ATMs), or any other devices. For example, a player may make a series of motions using their hand. These motions may be received by the player's wristband. The wristband may interpret these motions as commands to the ATM. The wristband may transmit these commands to the ATM. The ATM may then act in accordance with these commands, for example, by paying cash to the player.

[0043] (Wristband for 2D control) In various embodiments, the player may move their hand or arm within a single plane. Such motion may cause the cursor to move similarly on the screen. For example, if the player moves their hand first in one direction and then in the opposite direction, the cursor will also move first in one direction and then in the opposite direction. The player may rest their arm on a flat surface, such as the surface of a table. The player may rotate their hand on the surface of the table, thereby moving their hand in two dimensions. Thus, the wristband may be used to control the position of the cursor on a screen, such as the screen of a stationary game device, a portable game device, or other device.

[0044] (The string provides force feedback.) In various embodiments, the stationary game device may include a string, cable, wire, or other similar component. This string may be wound around a wheel, a central rod, spindle, shaft, or other device. The game device may include a motor for rotating this wheel. Rotation of this wheel in one direction releases more string, while rotation of this wheel in the other direction pulls the string in.

[0045] In various embodiments, the player may attach the end of the string to a wristband. Depending on the event in the game, the game device may either pull the string in or release more of it. This may have the effect of pulling and releasing the player's wrist. This may provide the player with tactile feedback. In some embodiments, the player may intentionally pull the string to generate commands in the game. For example, the player may pull the string outwards to spin the reels of a slot machine game. The faster or harder the player pulls the string, the faster the reels may spin.

[0046] (Distinguishing signals from multiple wristbands) In various embodiments, the game device may detect a signal from the wristband. The wristband may transmit an identifier to the player, thereby allowing the game device to recognize the player's identity. In various embodiments, when one game device detects a signal from the wristband, other game devices may also detect the same signal. Therefore, in various embodiments, the game device may determine whether it was the player's intention to communicate with it or with a different game device.

[0047] In various embodiments, the game device may recognize that someone is playing it. For example, the game device may detect an actual button press, or a player tracking card may be inserted, or currency may be inserted. Simultaneously, the game device may detect a signal from a wristband. The game device may display a message or otherwise ask the player currently playing the machine whether they are the person from whom the wristband signal was received. The game device may recognize the player's identity from the wristband signal and therefore display the player's name to the player physically present in the game device. Once the physically present player recognizes their own name, the player may confirm that the game device has indeed received the wristband signal from them. The game device may then allow the player to proceed with playing the game using motion controls.

[0048] In various embodiments, the game device may recognize that a wristband is nearby and that the game device is being played by a physically present player. Thus, the game may be easily started, for example, by physically pressing a button. The game device may then ask the physically present player whether they are the same player indicated by the signal received from the wristband. If the physically present player answers affirmatively, the game device may ask the player whether they wish to proceed using motion controls.

[0049] In various embodiments, the game device may distinguish between multiple signals coming from different wristbands as follows: Each wristband may be associated with a unique identifier. Each wristband may broadcast its own unique identifier. The game device may ask a physically present player which identifier corresponds to the player's wristband. In some embodiments, the game device may ask the player to input the identifier of the player's wristband. If this identifier matches the identifier of a signal received from one of the wristbands, the game device may then respond only to signals received from that wristband.

[0050] In various embodiments, the game device may ask the player to bring their wristband close to a reader. This reader may be an optical reader, an RFID reader, a magnetic stripe reader, or any other reader. In this way, the signal physically belonging to the player at the game device may be the clearly strongest signal received by the game device. The game device may then allow the player to physically use their wristband to proceed with the game. The player may then use some motion controls, or the player may use motion controls on the game device for each command.

[0051] (Reference light for fixed game machines) In various embodiments, a stationary game device may include one or more lights, beacons, transmitters, audio speakers, or other light-emitting devices. For example, a stationary game device may include two bright lights placed on top of the game device. These light-emitting devices may serve as reference points for a portable game device and / or a wristband. The wristband may, for example, detect light or other signals from the two light-emitting devices on the game device. The bracelet may use these two light-emitting devices as a fixed reference frame for measuring its own orientation. For example, if these two light-emitting devices appear side by side from the viewpoint of the wristband, the wristband may determine that its orientation is normal. However, if the two light-emitting devices appear one above the other, the wristband may assume that it has rotated 90°. In various embodiments, the light-emitting devices may output signals of the same type, for example, light of the same wavelength and amplitude. In some embodiments, different light-emitting devices may output different signals. This may allow the wristband or portable game device to distinguish one light-emitting device from the other in all orientations, thereby determining its own orientation more accurately. In various embodiments, a stationary game machine may have more than two light-emitting devices. For example, a stationary game machine may have three, four, or five light-emitting devices. In various embodiments, the light-emitting devices may be placed in locations other than directly above the stationary game machine. For example, the light-emitting devices may be placed on the ceiling or walls.

[0052] In various embodiments, the light-emitting device may emit light of a specific frequency. The light-emitting device may emit red light, green light, infrared light, or light of several other frequencies. The light-emitting device may emit light of multiple frequencies. For example, the light-emitting device may emit white light. The light-emitting device may also emit sound.

[0053] The wristband and / or portable game device may include sensors, cameras, microphones, or other detectors for detecting the output of a light-emitting device. For example, the wristband may include a camera that can detect light from a light-emitting device on the game device. Based on the position of the light-emitting device in the image captured by the wristband's camera, the wristband may determine its own orientation.

[0054] In various embodiments, the game device does not necessarily have a dedicated light-emitting device for detection by a wristband or portable game device. However, the wristband or portable game device may detect game device-specific features. For example, the game device may have a candle on top that is intended to light up when a casino participant is called to the game device (e.g., when a player of the game device wins a stake). A sensor in the wristband or portable game device may recognize an image of the candle. For example, the wristband may include a camera. This camera may capture an image and attempt to match a portion of the image with a pre-stored image of the candle on the game device. Based on the orientation of the candle from the captured image relative to the orientation of the candle in the stored reference image, the wristband may determine its own orientation. For example, if the captured image appears to be a 90° rotated version of the reference image, the wristband may assume that it has been rotated 90°.

[0055] In various embodiments, sensors within a portable game device or wristband may detect other features of a stationary game device. The sensors may detect a paytable, screen, handle, bet button, coin tray, image on the game device housing, betting meter, or any other feature of the game device. For any feature, the wristband or portable game device may store a reference image or reference signal. To detect or interpret a feature, the wristband or portable game device may capture an image and attempt to match a portion of that image to one or more reference images. In the matching process, the wristband or portable game device may manipulate the captured image, adjusting its size or orientation to better match the reference images. If a match exists (e.g., a portion of the captured image matches a reference image on a coin tray), the wristband or portable game device may determine the degree of rotation of the captured image that needed to be matched. This degree of rotation may then indicate the amount the wristband or portable game device was rotated.

[0056] In various embodiments, the game device may track the motion of a wristband or portable game device. The wristband may include beacons or light-emitting devices such as infrared light-emitting devices, light-emitting diodes, or audio speakers. The wristband may include two or more light-emitting devices. The game device may include detectors such as cameras, microphones, or antennas. The game device may determine the position or relative position of the light-emitting devices on the wristband. For example, in a vertically upright position, two light-emitting devices on the wristband may appear side by side. When the wristband is rotated 90°, one light-emitting device may appear above the other. Thus, based on the relative positions of the two light-emitting devices on the wristband, the game device may be able to determine the orientation of the wristband. The apparent distance between the two light-emitting devices on the wristband may also provide an indication of the distance of the wristband itself from the game device. For example, if the two light-emitting devices on the wristband appear close to each other, it may be assumed that the wristband is far away. On the other hand, if the two light-emitting devices on the wristband appear far apart from each other (at least relatively speaking), it may be assumed that the wristband is close. A game device (e.g., a slot machine; e.g., a video poker machine) may confirm commands intended by a player by tracking the motion of a wristband or portable game device. The game device may execute those commands in the game it is playing. The game device may also transmit those commands to another device, e.g., another stationary game device, or e.g., a portable game device.

[0057] (Screen guidance for motion control) In various embodiments, a game device, such as a stationary game device, may provide the player with instructions on how to use motion control. The instructions may show one or more available commands that the player can give. For example, the game device may list commands for: (a) starting a game; (b) making a selection in a bonus round; (c) selecting a card to discard in a video poker game; (d) selecting whether to hit or stand in a blackjack game; (e) selecting a payline to bet on; or taking any other action in the game or elsewhere. The game device may also provide instructions on how to issue commands. The game device may show what motion is required to issue a command. The game device may show a short video or animation of people making motions with their hands. Thus, after a potential command, the player may see a short video clip of people moving the player's arm in a particular way. This video clip may be continuously repeated or played on request (e.g., in response to player contact). The motion to be performed to issue a command may be described in text form, such as "Move your hand twice to the right, then once up." Instructions on how to use motion controls may be presented in many different formats.

[0058] In some embodiments, a person may be guided by instructions or given the opportunity to practice performing motions. For example, instructions for performing a motion corresponding to the "Start Game" command may be performed in the form of a video clip. In other words, an animation of a person performing a particular motion may be shown on the game device's display screen. The player may be instructed to repeat the motion with their own wristband. The player may be instructed to follow a video of the motion being performed. If the game device recognizes the motion, it may ask the player to perform the motion for the next instruction. If the game device does not recognize the motion performed by the player (for example, if the player performs an incorrect motion), the game device may ask the player to repeat the motion until the player masters the correct motion.

[0059] In various embodiments, when a player is playing a game on a game device (for example, a slot machine) and the player is making motions to issue a command, the game device may provide feedback on how the game device has interpreted the player's motions. For example, the game device may display a text message, "You have motioned to start a new game."

[0060] (Time window for performing motion) In various embodiments, there may be a finite time window during which a game device (e.g., a stationary game device) will accept motion commands. For example, there may be a 10-second window during which the game device will accept motion commands. During other times, the player may perform motions, but these may not necessarily be registered as commands. This may give the player some freedom to perform motions unrelated to the game (e.g., hand gestures in conversation) during times outside of the window in which commands may be registered. The time window for performing motion commands may open and close periodically. For example, the window may be open for 10 seconds, then closed for 20 seconds, then open again for 10 seconds, and so on. If a person performs a first motion command during the time window, this time window may be extended. For example, extending the time window may allow the person to complete the entire game before the window for performing motion commands closes. In some embodiments, the time window for performing motion commands may continue as long as the game is in progress. In some embodiments, the time window for performing motion commands may continue for a predetermined time after the last motion command produced by the player. This may allow the player to continue making motion commands for as long as the player desires. In some embodiments, there may be a warning or other indicator that the game device (e.g., a stationary game device; e.g., a portable game device) is accepting a motion command. For example, an indicator light on the game device may light up, or the indicator light may change from one color to another. So, for example, the light may be blue when the game device is accepting a motion command, and red when the game device is not accepting a motion command. In some embodiments, the player may turn motion control on or off. For example, the player may instruct the game device to enter a state where it is accepting motion commands, or instruct the game device to ignore motion commands.The player may need to physically touch the game device to either turn a motion command on or off. In some embodiments, when the game device does not accept a motion command, it may still respond to a motion command that instructs the game device to accept other motion commands again. For example, the game device may then become accepting of motion commands again.

[0061] In various embodiments, the first set of motions may correspond to moving a cursor, mouse pointer, or other indicator. The second set of motions may correspond to making a selection. For example, if the cursor is resting over an image of a card or button, making a motion in the second set of motions may correspond to selecting that card (e.g., selecting to discard that card) or pressing that button. Motions from the second set of motions may be used, for example, to select a bet amount, to select a payline, to select a decision from a decision menu, or to make any other selection. Motions from the first set of motions may position the cursor for a later selection, but have not yet handed the player over to an action on a course of action. In some embodiments, forward and reverse motions (e.g., from the player's perspective) may correspond to the second set of motions, for example, making a selection. Motions in other directions (e.g., upward, downward, left, right) may correspond to motions from the first set of motions, for example, positioning the cursor.

[0062] In various embodiments, the player may receive visual feedback as they make motions. The cursor may follow a trajectory created by the player's wristband on the screen of the game device (e.g., a stationary game device; e.g., a portable game device) as the player's hand moves. To make a particular command, the player may need to keep the cursor within a specific boundary. For example, a boundary consisting of two concentric circles may be displayed on the display screen of the game device. The player may need to create a circle with the cursor while keeping the cursor in a part that is outside the inner circle but inside the outer circle (i.e., between the two circles). In some embodiments, points or dots may exist on the screen. The player may need to make motions such that the cursor on the screen moves between two dots. In some embodiments, several pairs of dots may exist. To issue a command, the player must move the cursor between various pairs of dots in some specific order. Different commands may require moving the cursor between different pairs of dots in a different order.

[0063] In various embodiments, the player may perform a motion command to position the cursor over a button. Further motion commands may be performed to select a button. Different buttons may correspond to different commands or actions within the game. Therefore, by performing a motion to position the cursor over the appropriate button, the player may perform a desired command within the game.

[0064] (The wristband senses the tension in the wrist muscles in the form of a grasping motion.) In various embodiments, the player wristband may be equipped with a strain gauge. The wristband may be manufactured from a flexible material such as rubber. The wristband can fit snugly around the player's wrist. When the player closes their fist, they may tense certain wrist muscles. This increases the circumference of the player's wrist, which can place additional stress on the wristband. The strain gauge can sense this additional strain on the wristband. The strain gauge can signal to a processor in the wristband indicating the detected strain. The strain gauge can also signal to another device, such as a portable gaming device, a stationary gaming device, or a casino server, via an antenna or other transmitter.

[0065] In various embodiments, the wristband may have one or more pressure sensors on its inner surface, for example, on the surface that contacts the player's wrist. These pressure sensors can sense pressure from the player's wrist and indicate possible tension in the wrist or flexion of the wrist muscles.

[0066] In various embodiments, the wristband may have a temperature sensor. The sensor can detect an increase in temperature at the wrist, resulting from increased blood flow and / or more rapid burning of energy in the wrist muscles. Readings from these sensors can correspond to the tension in the player's own wrist, such as when the player performs a grasping motion.

[0067] In various embodiments, the electrical activity of nerves or muscles in the wrist may change depending on whether the muscles are tensed or relaxed. A sensor in a wristband, such as an antenna, can sense the electrical activity in the wrist and interpret the electrical activity as an indicator of whether the wrist muscles are tensed or relaxed.

[0068] In various embodiments, tension in the wrist muscles may be interpreted as a command in a game. In various embodiments, tension in the wrist muscles may be interpreted as a button selection or a choice from multiple options. In various embodiments, tension in the wrist muscles may substantially correspond to grasping something in a game. For example, in a bonus round, a game character can grasp the knob of one of three doors to open it. Since tension in the wrist muscles can be triggered by the player actually performing a grasping motion (for example, in the real world), the player can use the grasping motion as an intuitive way of making a selection or grasping something in the game. Thus, for example, the player can move a cursor by linear displacement of the hand and select something by performing a grasping motion.

[0069] In various embodiments, sensors or detectors can detect grasping motions or other hand or wrist motions, even if such sensors are not located within a wristband. For example, a camera may capture the player's hand motion. An image processing algorithm may be used to recognize that the motion is being made by the player's hand. These motions may be translated into commands in a game.

[0070] Thad Starner, Joshua Weaver, and Alex Pentland of the Massachusetts Institute of Technology have developed a camera-based system for recognizing American Sign Language. The system is described in a paper titled "Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video."

[0071] (Slot machine receiver) In various embodiments, a game device such as a slot machine may include a Bluetooth transceiver. The transceiver may be integrated into the device. The transceiver may also take the form of a Bluetooth dongle that can be connected to a Universal Serial Bus (USB) port of the game device. In various embodiments, the game device may include a Wi-Fi transceiver. The game device may send and receive messages to and from a wristband or portable game device using Bluetooth, Wi-Fi, or any other communication protocol.

[0072] (Component of the message from the wristband) The data content of a signal from a wristband may include one or more components. The signal may be understood to always include these components in a specific order, for example. For example, the first three bits of the signal may indicate the start of a new message. The next four bits may indicate the type of device giving the transmission (e.g., a wristband; e.g., a portable game device). The next thirty bits may give an identifier for the wristband. The next hundred bits of the signal may give the player name. The next twenty bits may give a command. The last ten bits may indicate that the signal has ended. In some embodiments, the signal may include one or more of the following parts or regions: (a) a region indicating the start of the signal; (b) a region indicating the type of device transmitting the signal; (c) a region indicating the intended reception of the signal (e.g., a unique identifier for a game device; e.g., an identifier for a casino server); (d) a region indicating a player identifier; (e) a region indicating a device identifier (e.g., a unique identifier for a particular device transmitting the signal); (f) a region indicating the end of the signal; (g) a region indicating a player name; (h) a region indicating a command used in the game; (i) a region indicating a name identifier (an identifier for the game to which the command applies); (j) a region containing one or more error checks; and any other regions.

[0073] (Confirmation of the presence and identification of players in fixed game machines) In various embodiments, the wristband may transmit a signal, which may be received by a fixed gaming device. The signal may include an identifier for the wristband. The gaming device may transmit the wristband identifier to a casino server. The casino server can then look up the name of the player signing the wristband (e.g., the player currently using the wristband). The casino server may transmit the player's name to the gaming device. In some embodiments, the signal from the wristband may include a player identifier. The gaming device may transmit the player identifier to the casino server. The casino server may then transmit the player's name back to the gaming device. At any event, the gaming device may determine the player's name. The gaming device may display a message indicating the player's name. The message may be a greeting; for example, the message may say, "Hello, Sarah Jones!" The message may also ask the player to confirm their identity. A player can verify their identity by answering a secret question, by providing biometric authentication (e.g., fingerprints), by inserting a player tracking card, by inserting a credit card, by inserting a bank card, by inserting a driver's license, by flashing one of the aforementioned cards in front of a camera, or by any other means. In various embodiments, a player can verify their identity through physical contact with the game device. For example, a player can answer a secret question by physically touching letters on the game device's touchscreen and writing the answer in that manner. If a player verifies their identity through physical contact with the game device, the game device can adequately ensure that it is not controlled based on motion from someone other than the person sitting in it, or by other wireless commands.

[0074] (A prominent screen for playing using motion controls only) In various embodiments, a casino or other establishment may be equipped with a large display screen. The screen may display a game. The screen may show the progress and action of a game, such as a slot machine game or a video poker game. Electrical or other devices associated with the screen may cause the screen to receive motion input for playing the game. For example, there may be an antenna for receiving signals from a player's wristband, or a camera for reading the player's motion commands. A processor or other device may calculate or determine game events or game results. Players may be given value or currency to gamble by inserting cashless game tickets. Thus, associated with the screen may be a device for inserting and removing tickets for receiving and distributing cashless game slips (game tickets).

[0075] The player can play the game on a large display screen. The player can issue commands in the game using motion controls. For example, the player's wristband can detect motions made by the player's hands. The motion instructions may be sent to the large display screen. The large display screen can then advance the game course as instructed by the player's commands.

[0076] In various embodiments, a motion-controlled game using a large display screen may be located at each end of two or more rows of a slot machine. For example, a large display screen featuring a game using motion control may be present at the end of each row of a slot machine or other gaming device. Such a game can be seen by anyone in the row of slot machines. In this way, people playing the slot machine can see the game they are playing on the large screen and try out motion control itself.

[0077] (Toggle button on the wristwatch to activate or deactivate the function) In various embodiments, the wristband may include a switch, button, toggle, or other device for selecting between two or more states. The switch may be used to enable or disable motion control. Thus, when the switch is in one position, the player wearing the wristband can use motion control to control actions in the game. When the switch is in another position, the player cannot use motion control to control actions in the game. If the player does not wish to play the game for a short time, they can press the switch to disable motion control. The player can then make wrist gestures without worrying that the gestures will affect the outcome of the game. If the player wishes to play the game again and use the game's motion control, they can press the switch again to enable motion control.

[0078] In various embodiments, the player may use switches or other devices to activate or deactivate other features of the wristband. The player may activate or deactivate haptic feedback. For example, using a switch in a certain position, the wristband may provide the player with force feedback or haptic feedback. If the switch is in a different position, the wristband may not provide such feedback. The player may wish to deactivate haptic feedback, for example, to conserve battery power in the wristband. In some embodiments, the player may turn sound on or off. For example, in at least one state, the wristband may emit an audio signal. The audio signal may be related to the game (for example, victory music may be emitted from the wristband if the player wins). The audio signal may be related to the player's position. For example, if the player enters a restricted area where the game is not permitted, the wristband may emit an audio signal. The audio signal may be related to the account balance. For example, if the player's account balance becomes zero, the wristband may emit an audio signal. Other reasons may exist for audio signals emitted by the wristband.

[0079] In various embodiments, the wristband may comprise one or more buttons, one or more sensors, one or more piezoelectric sensors, a batter, a transmitter, a receiver, and an onboard processor. The buttons allow the player to change the environment or state of the wristband (e.g., turn sound on or off). The buttons allow the player to be given commands about the game, and such commands do not have to be motion-based. The sensors may comprise motion sensors such as accelerometers or gyroscopes. The sensors may comprise position sensors such as GPS sensors. The sensors may comprise temperature sensors, pressure sensors, strain gauges, microphones, light sensors, or any other sensors. The sensors can perform a variety of functions. Since the sensors can detect motion, such motion can be converted into commands. Since the sensors can sense the player's position, it can be communicated whether the player is in an permitted game area. Sensors may sense the player's muscle tension or electrical activity, for example, to drive motion commands. The transmitter may be used to communicate with another device such as a fixed game device, a portable game device, or a casino server. The receiver may receive communications from another device, such as a portable gaming device, a fixed gaming device, or a casino server. Communications received by the wristband may reprogram the wristband. Such communications may, for example, give commands to the wristband. For example, communications received by the wristband may instruct the wristband to stop when the player's account balance becomes zero.

[0080] (Holding hands) In various embodiments, the wristbands of two players may interact. This interaction may occur when the wristbands come close to each other. For example, if two players shake hands with their wristband-wearing hands, the two wristbands can interact.

[0081] In various embodiments, during the interaction, the wristband of the first player can receive information from the wristband of the second player. The wristband of the second player can receive information from the wristband of the first player.

[0082] In various embodiments, the portable game device of the second player can receive information from the wristband of the first player.

[0083] In various embodiments, bets can be made or decided by two players shaking hands. Technically, in some embodiments, the bet can be made when the wristbands of the two players are within a predetermined distance (e.g., 5 inches (12.7 cm)) of each other for a predetermined time (e.g., 5 seconds). In some embodiments, the bet can be made when the wristbands are within a predetermined distance of each other for a predetermined time and there is a motion of gripping one or both wristbands. The gripping motion corresponds to shaking hands. If the wristbands are gripping hands, they may transmit information to each other about the timing of the gripping motion to ensure that the wristbands are gripping at the same time. In various embodiments, the first player may predetermine the timing of the bet using a fixed game device or other device. For example, the first player can place a bet so that if the roulette wheel spins and lands on black, the first player wins $1 from the second player, and if the roulette wheel spins and lands on red, the second player wins $1 from the first player. Once a bet is specified, the first player only needs to find the second player who will hold hands to decide on the bet. In various embodiments, the first player is allowed to mischaracterize the duration of the bet for the second player. Thus, in various embodiments, the first player can predetermine only fair bets (e.g., bets where both sides have an equal chance of winning and / or bets where both sides are expected to win equally and / or bets where both sides are expected to win and lose zero). In various embodiments, when players hold hands to place a bet, the duration of the bet may be displayed on one or both of the players' portable game devices. Each player may have a window of time (e.g., 30 seconds) to cancel the bet. To cancel a bet, a player can, for example, press the "cancel" button on their portable game device.If no players cancel their bets, a result can be drawn, and that bet can be decided by one side or the other.

[0084] In various embodiments, the first wristband can detect the proximity of another wristband. The wristband may be Bluetooth-enabled so that it can detect the proximity of another wristband by transmitting via the Bluetooth protocol. In various embodiments, the wristband may be programmed or configured to send and receive signals of other protocols, such as Wi-Fi.

[0085] In various embodiments, two or more players may hold hands to place bets with each other. The winning player may depend on the outcome of several games, such as games played or simulated by the game device. In some embodiments, in order to decide on a bet, the two players must be in close proximity to a game device, such as a fixed game device. For example, in order to proceed with a bet, the two players must be standing in front of a slot machine. Players may be required to be within a predetermined distance of a particular game device, such as within two feet. Wristbands of one or two players may communicate with the game device to indicate that the players have agreed to the bet. One or both wristbands may communicate with the game device for the duration of the bet, such as the game on which the bet depends. The game device can then play an appropriate game to fulfill the bet. For example, if the bet is a video poker game, the game device can play a video poker game. If the bet is a blackjack game, the game device can play a blackjack game. In various embodiments, the wristbands may communicate with the game device in which the player will win under either of the following circumstances. For example, a wristband can communicate with a game device, where if the house wins the game of blackjack, "Joe Smith" wins, and if the player wins the game of blackjack, "Jane Smith" wins. In this case, the player can be said to be a virtual player simulated by the game device. The game device can play a basic strategy or an optimal strategy on behalf of the virtual player. In some embodiments, two players betting on a game can play the game against each other using one or more game devices. Players can instruct the game device on strategy decisions. For example, if two players bet on a game of blackjack, the players can efficiently agree to play the game of blackjack against each other. Two players can play on a specific game device. During the course of the game, players can give decisions about the game.Players can make decisions by physically pressing buttons on the game device or by physically interacting with the game device. Players can also make decisions using motion controls, for example, by using their wristbands.

[0086] (Incentives for holding hands) In various embodiments, there may be an incentive to shake hands with people. A person's wristband can track the number of times a person shakes hands with someone else, and / or the number of people a person shakes hands with. In some embodiments, after shaking hands with each other, a player's wristband can send a record of the handshake or other instructions to the casino server. The wristband can also transmit identifiers about other players or other wristbands that the player has come into contact with. The casino server and / or the player's wristband can track the number of times a player shakes hands with other players. The casino server and / or the player's wristband can also track the names or identifications of other players that the player has shaken hands with. In various embodiments, the player who shakes hands with the most other players over a certain period (e.g., one day) may win a prize, such as $1,000.

[0087] In some embodiments, the mixer may be held at a casino or related estate or any other site. The mixer may be an opportunity for singles to meet, an opportunity for business people to connect, an opportunity for scientists to exchange ideas with colleagues, or any other type of mixer. During the mixer, people may shake hands with each other. People's wristbands may automatically exchange information including name, contact information, email address, telephone number, biographical information, photograph, credit information, place of residence, age, gender, marital status, or any other information or any other information that may be appropriate for the situation.

[0088] The wristbands of those participating in the mixer can transmit information to the casino server or other device information about the people they shook hands with or came into contact with. Those who were in the mixer can later take notes on a website to view a summary list of the people they met. The website may include contact information about the people. In some embodiments, contact information is not provided. Rather, a person must select the people they wish to contact. If one person selects another, and the other selects them, the website can later provide both of them with each other's contact information.

[0089] In some embodiments, during a handshake, a person's wristband can transmit their information (e.g., contact information) to another person's portable device (e.g., portable game device; e.g., personal digital assistant; e.g., mobile phone). Thus, at the end of the mixer, a person can store information on their portable device about other people they have met during the mixer.

[0090] In various embodiments, at the end of the mixer, a person can see images of the people they have met in the mixer. By viewing the images, they can recall memories of the people they have met. The person can then select the people they are interested in further contacting. They may then be given information about those people. In some embodiments, the person may only be given information about people they are interested in further contacting if they are also interested in the possibility of further contact with them.

[0091] In various embodiments, the mixer is kept in a bar, restaurant, lounge, gym, swimming pool, gambling floor, or any other lounge.

[0092] (Payment is made by shaking hands.) In various embodiments, a player may make a payment by shaking hands. A player may pay for drinks, foot items, retail products, or any other items by shaking hands. In some embodiments, a casino employee or retail employee may possess a wristband. When an employee shakes hands with a person (e.g., a customer; e.g., a player), the employee's wristband may receive a communication from the player's wristband. The communication may include information about the player, such as the player's name, identifier, credit card identifier, financial account identifier, or any other information. The employee's wristband may communicate the player's financial account identifier and other identifying information about the player to a sales terminal, retail server, casino server, or any other device. The player can then charge the wristband for purchases via a credit card network or other financial network.

[0093] By shaking hands with a casino employee, retail employee, salesperson, or other person, the player may have a limited time to evaluate and cancel a transaction. For example, the player's wristband may also store the details of a transaction after shaking hands with a salesperson. The details of the transaction may include the purchase price, product, and delivery method. The player can bring their wristband close to a portable or stationary game device. The wristband can transfer the details of the transaction to the portable or stationary game device. The portable or stationary game device can then display the details of the transaction about the player. The player can evaluate them and decide whether to cancel them. If the player wishes to cancel, in some embodiments, the player can press a button or screen area on the portable or stationary game device. The player may also be required to return to the place where they bought the product and returned it.

[0094] In various embodiments, a player may bring their wristband close to a reader as a way to pay for a commercial transaction. The player may also touch a pad with their wristband. For example, a player may place their hand on a pad to pay for a drink. The pad may be equipped with an antenna or other type of receiver to detect a signal from the wristband. The detected signal may include a financial account identifier.

[0095] In various embodiments, a player can use their game credit balance to pay for purchases or other commercial transactions. A player may have an account for game credits that is stored and tracked on a casino server. If a player holds their wristband near a pad or reader to make a purchase, the reader can verify on the casino server whether the player has sufficient account balance to complete the purchase. In various embodiments, if the player has sufficient account balance, the pad or reader may give a first instruction; if the player does not have sufficient account balance, it may give a second instruction. The first instruction may be, for example, a green light. The second instruction may be, for example, a red light.

[0096] (The wristband will be in a state where the clasp is undone.) In various embodiments, if the wristband separates from the player (for example, if the clasp is undone), an alert may be sent to the casino server. This alert may indicate to the casino server that the wristband is no longer surrounding the player's wrist. In various embodiments, once the wristband is removed, it may cease to function for gaming purposes. For example, the wristband can no longer provide motion control. The wristband may also cease to transmit the player identifier to the portable gaming device. Therefore, the player's portable gaming device can no longer engage the player in gambling activities. Various other functions of the wristband may also cease once the wristband is removed.

[0097] In various embodiments, if a player wishes to restore the various functions of the wristband, the player may visit a special service area of ​​the casino, such as a casino desk. There, a casino employee may return the wristband to the player. The casino employee may send a special code to the wristband to reactivate it. The casino employee may also check the player's identity before reapplying the wristband, for example, by requesting fingerprints or a driver's license.

[0098] In various embodiments, the wristband includes one or more sensors for determining whether the wristband is separated from the player, whether the clasp has been undone, or otherwise tampered with or removed. For example, the sensors may include an electrical circuit surrounding the wristband. If the wristband is detached, the circuit may be damaged.

[0099] In various embodiments, a wristband or portable gaming device may rely on continuous or periodic communication with a casino server to function. If the wristband or portable gaming device loses contact with the casino server, it may cease to function. In various embodiments, the wristband may communicate with the server periodically. Inputs received by the wristband from the player cannot be executed until the next communication is received from the server. For example, if a player moves their hand to create a command, the wristband may store a record of the motion and / or store the command corresponding to the motion. However, the wristband cannot transmit that command to another device, for example, the portable gaming device or gaming device that the player might be playing. Rather, the wristband may store the command until it receives a communication signal from the server again. In this way, the wristband can ensure that no command or game command is executed while the wristband is unable to communicate with the casino server. In some embodiments, the wristband may accumulate inputs received from the player. However, if the wristband does not receive communication from the casino server within a predetermined period in which it receives inputs, the wristband may discard those inputs. In this way, the player will not be surprised later when a large number of saved or pre-loaded commands are executed simultaneously. In various embodiments, a player entering an elevator may be unable to play for a while because communication between their bracelet and the casino server may be interrupted.

[0100] In various embodiments, instead of a wristband that ceases functioning when opened or the clasp is released, the wristband may continue broadcasting "I am open" to the server until the server confirms it. There may be a certain period of time between the time the wristband is opened and when it attempts to inform the server that it is open. Then, there may be a certain period of time between when it receives confirmation from the server and when it stops broadcasting. After the wristband is opened, it may no longer be able to permit some functions (e.g., payments to be made using the wristband), but other functions (e.g., motion control) may still be permitted. In other words, in various embodiments, some functions will not work when the clasp is released or otherwise when the wristband is removed.

[0101] (The wristband and portable gaming device can replicate each other's functions.) In various embodiments, any motion command that can be created by a wristband can also be created by a portable game device. For example, just as a wristband may be equipped with sensors that detect acceleration, changes in direction, displacement, and any other motion, a portable game device may also be equipped with such sensors. Similar to a wristband, a portable game device may be equipped with a processor that reads signals from motion sensors in the portable game device and interprets such motion as a command to be used in a game, or as any other command. In various embodiments, any command that can be created via a portable game device can also be created using a list. In various embodiments, a wristband may detect motion created by a player and transmit instructions for that motion to a portable game device. The portable game device may interpret that motion as a command in a game, or as any other command. In various embodiments, a portable game device may detect motion and transmit that motion to a wristband. The wristband may interpret that motion as, for example, a command in a game. The wristband may then transmit instructions for that command to a stationary game device. In various embodiments, any signals or alerts broadcast by a portable game device based on its position may be similar to those broadcast by a wristband based on its position. For example, if a player wanders outside the designated game area, the portable game device or wristband may detect the player's position and issue an audible alert for the player. In various embodiments, any haptic feedback that may be provided by the wristband may also be provided by the portable game device. In various embodiments, any information received, determined, or detected by the wristband may be communicated to the portable game device, for example, via wireless communication.

[0102] The following are embodiments and not claims. Various embodiments include: A. Methods including the following: Receiving a first radio signal from a first device; Receiving a second radio signal from a second device; Determining a first player identifier from a first radio signal; Determining a second player identifier from a second radio signal; Displaying a message to the player prompting them to identify themselves; Receiving instructions for a third player identifier via haptic input; To determine that the third player identifier matches the first player identifier; Receiving a third radio signal from a first device; Interpreting a third radio signal as a command in a gambling game; and Executing a command in a gambling game. Executing a command may include fulfilling a command, following a command, acting in accordance with a command, and / or acting in accordance with a command. B. The method of Embodiment A, wherein the first device is one of (a) a wristband; (b) a watch; (c) a bracelet; (d) an armband; and (e) a portable game device. C. The method of Embodiment A, wherein determining a first player identifier from a first radio signal includes determining the name of a first player from a first radio signal. For example, the first radio signal may encode a player name. In some embodiments, the player name may be found in a database that associates the player with other player identifiers having a player name (e.g., player tracking card number). D. The method of Embodiment A, wherein the first player identifier and the second player identifier correspond to different players. E. Receiving instructions for a third player identifier via tactile input is a method of Embodiment A, which includes receiving instructions for a third player identifier being entered using a button. For example, someone may enter a third player identifier by physically pressing a button (e.g., a letter key) on the game device. F. Receiving instructions for a third player identifier via haptic input includes receiving instructions for a third player identifier being input using a joystick, according to Embodiment A. G. Receiving instructions for a third player identifier via haptic input includes receiving instructions for a third player identifier being entered using a touchscreen, according to Embodiment A. H. Receiving instructions for a third player identifier via haptic input is a method of Embodiment A, which includes receiving instructions for a third player identifier being entered using a trackball. I. The method of Embodiment A, wherein a third radio signal encodes a set of motions created by the first device. For example, the third radio signal may include a set of numbers representing position, velocity, acceleration, displacement, angular displacement, or other components of motion. The numbers may be understood to represent degrees, centimeters, or other units of measurement. In some embodiments, the third radio signal may include an identifier for one of a set of recognized motions (e.g., "motion F"; e.g., "zigzag motion"). J. The method of Embodiment A, wherein interpreting the third radio signal includes interpreting the third radio signal as a command to discard a card in a video poker game. K. The method of Embodiment A, wherein interpreting the third radio signal includes interpreting the third radio signal as a command to start a slot machine game. L. Equipment including the following: A band formed in a ring shape; Power supply attached to the band; Motion sensor attached to the band; Electromagnetic transmitter attached to the band; Audio speaker attached to the band; A tactile transducer attached to a band; A processor mounted on the band; and An electromagnetic receiver attached to the band. The band may be a metal band, a rubber band, a chain band, a cloth band, a leather band, or any other type of band. In some embodiments, the band may be formed into a loop by fastening its two ends together. In some embodiments, the band is always in a loop shape except for unintentional breaks or tears. M. The device of Embodiment L, wherein the haptic transducer is operable to generate vibrations in response to electrical signals from a processor. For example, the processor may instruct the haptic transducer to vibrate when a jackpot is won in a game being played by the wearer of the device. N. The device of embodiment L, in which the motion sensor is an accelerometer. O. The device of embodiment L in which the processor can be operated as follows: The first electrical signal is received from the motion sensor; Based on a first electrical signal, a first command for a first gambling game is determined; Send the first command to the electromagnetic transmitter; The electromagnetic transmitter is instructed to send the first command to the first game device. Therefore, in various embodiments, the device may detect the player's motion and interpret that motion as a command in a gambling game such as a slot machine game, a video poker game, a blackjack game, or any other game. The device then transmits the command to a game device, such as a slot machine or a portable game device, so that the command can be executed in the game. P. The device of Embodiment L in which the processor can be operated as follows: Receive instructions received wirelessly by an electromagnetic receiver from the electromagnetic receiver; A second electrical signal is received from the motion sensor; Based on the second electrical signal, follow the instructions to determine the second command for the second gambling game; Send the second command to the electromagnetic transmitter; and The electromagnetic transmitter is instructed to send a second command to the game device. Q. Including a switch attached to the band, The apparatus of embodiment L, wherein the switch has two stable positions, and the processor is operable to detect the position of the switch and instruct an electromagnetic transmitter to transmit a signal only when the switch is in the first of the two stable positions. In various embodiments, the player may switch some or all aspects of the wristband on or off. The player may do this using a switch, button, other switching device, or other device. In one state of the switch, the wristband may transmit motions or commands to be used in the game. In another state of the switch, no such motions or commands may be transmitted. For example, the player may want to create motions without worrying about whether such motions may be counted in the game. The apparatus of embodiment L further comprises a piezoelectric sensor attached to the band. The piezoelectric sensor may detect flexion of the player's wrist muscles, for example, through the pressure the muscles exert on the wristband. S. Equipment including the following: A housing with a top surface parallel to the ground; A coin hopper placed inside the housing; A banknote authentication machine mounted on a housing; Display screen mounted on the housing; The processor is located inside the housing; A wireless receiver mounted in a housing; A wireless transmitter mounted in a housing; A first light source mounted on the top surface of the housing, operable to emit light of a first frequency; and A second light source mounted on the top surface of the housing at least one foot away from the first light source, configured to emit light of a second frequency different from the first frequency. The device may represent a game device. Its two light sources may provide a fixed reference point to which the wristband or portable game device can determine its own position or orientation. For example, the first light source may be green light and the second light source may be red light. The wristband may detect the two light sources by, for example, capturing an image containing the light sources, determining the apparent distance of the light sources in the image, and determining its own distance from the light source based on the known distance between the two light sources. The device of embodiment S in which the T processor can be operated as follows: To play gambling games; and The course of the gambling game is altered based on radio signals received by a wireless receiver. In various embodiments, altering the course of a gambling game may include taking one of two or more possible actions in the gambling game, such as selecting one or more possible cards to keep, or selecting one of two or more possible bets.

[0103] (Several haptic technologies) The Immersion Impulse Stick is a joystick that provides force feedback and is marketed for use in demanding environments such as arcades.

[0104] Immersion's VibeTonz® system is a system that can give mobile phones a tactile sensation. Such sensations can provide the feel of a simulated machine gun, the impact and decay of an explosion, or the feeling of a foot hitting a ball.

[0105] A "tactile interface device" provides a user of a sensory interface device with tactile sensations (tactile displays) in response to the user's interaction with the environment to which the tactile interface device is associated. "Tactile" refers to the sensation of touch; therefore, a tactile interface device generates sensations associated with touch, such as texture, force (e.g., friction, magnetic repulsion, or magnetic attraction), vibration, mass, density, viscosity, temperature, humidity, or any combination of such sensations. Tactile interface devices can be embodied in a wide variety of devices, including, for example, devices for transmitting force sensations and / or vibrational tactile sensations (e.g., styluses, movable arms, wheels, dials, rollers, sliders, or vibrating surfaces), devices for transmitting thermal sensations (e.g., thermally controlled surfaces or airflow), and devices for transmitting humidity sensations (e.g., humidly controlled surfaces or airflow). Tactile interface devices can be used in a wide variety of applications. For example, some joysticks and mice used with computers incorporate force feedback to provide tactile feedback to the joystick and mouse user. Some paging devices are adapted to vibrate when a paging signal is received. Some toys generate vibrations as part of the toy's interaction. These examples suggest the wide range of applications in which tactile interface devices can be used.

[0106] In conventional haptic interface devices, the characteristics of the haptic display experienced by the user are determined by a haptic model that associates the state of one or more aspects of the environment with the haptic sensations given to the user. The user interacts with the environment (directly or via the haptic model) through the environment interaction model using an environment interaction control device. The haptic model "interprets" the user interaction with the environment (based on information about the user interaction obtained from either the environment interaction model or the environment) and generates a haptic display corresponding to the haptic display device. The environment interaction model can also generate non-haptic displays (e.g., visual and / or auditory displays) on non-haptic display devices. However, non-haptic displays are not always necessary.

[0107] The magnitude of the change in tactile sensation per unit change in the state of one or more aspects of the environment is expressed herein as the “resolution” of the tactile display. For example, in a tactile interface device used for video browsing and / or video editing, a knob may be rotated to advance frames of a video recording, and a force opposite to the rotation of the knob may be applied to simulate a stop in the video recording at a predetermined transition from one video frame to the next. The resolution of the tactile display in the tactile interface device may be the frequency of stop occurrences in the video recording (e.g., the number of video frames between each stop). (It is also possible to define the resolution of the tactile display in such a tactile interface device with respect to the frequency of stop occurrences per unit duration of video obtained, as illustrated by the embodiments discussed further below.)

[0108] Outputs generated by tactile display devices may include, for example, texture, force (e.g., friction, magnetic repulsion, or magnetic attraction), vibration, mass, density, viscosity, temperature, humidity, or any combination of such sensations. When the environment is a visual and / or auditory recording, for example, force can be applied in opposition to the movement of a device embodying an environment interaction control device, and a tactile display device simulating a stopper as a transition is created from one video frame (or other relevant set of visual recording data) to the next. Furthermore, tactile models can reproduce various features of tactile sensation, such as inertia, damping, and / or compliance. Tactile display devices can utilize various devices to generate tactile displays. For example, devices for generating force and / or vibration tactile sensations can be used where appropriate for the desired tactile display, such devices include, for example, DC servo motors, sound coil motors, linear actuators, hydraulic actuators, pneumatic actuators, shape memory alloys (SMAs), and piezoelectric transducers. Where appropriate for the desired tactile display, a heating device may be used additionally or as an alternative, such as, for example, a thermoelectric module or a combination of a heater and a fan. Where appropriate for the desired tactile display, a humidifying device and / or humidifying material may be used additionally or as an alternative, such as, for example, a condenser, a sprayer, a moisture-permeable barrier, and anhydrous material.

[0109] Tactile display devices can be embodied, for example, by force-operated wheels, knobs, handles, or arms, heat sources and / or heat dissipators, or humidifiers and / or moisture absorbers.

[0110] Various devices actively respond to user input by providing tactile cues or responses to the user. A mobile phone's vibrator or pager is a good example. Other examples include input keys that provide a clicking sound when moved; keys or touchscreens that suddenly move or vibrate in the opposite direction to the input; and keys that suddenly move or vibrate perpendicular to the direction of the input depending on a transducer mounted on the device housing.

[0111] The display device and / or input mechanism such as a key may be configured to provide active haptic force feedback. An electromechanical transducer, such as a voice coil-based linear vibration motor, piezoelectric actuator, or piezoelectric vibrator, may be mechanically connected directly to the display device, and an electromechanical transducer such as a vibrator may be mechanically connected directly to the key.

[0112] In various embodiments, the haptic interface module is configured to output pulses of predetermined or user-defined amplitude and duration in response to receiving a trigger signal from the telephone processor. Alternatively, other interface logic (e.g., address decoding logic) is included between the digital signal bus and the haptic interface module. The telephone processor is programmed to trigger the haptic interface module in response to predetermined states, such as those determined by intelligent operation within the telephone processor. Optionally, triggering the haptic interface module can be selectively enabled or disabled according to user-editable configuration settings. The haptic interface module is connected to an electromechanical transducer. The electromechanical transducer is driven by the output of the haptic interface module.

[0113] More generally, an electromechanical transducer is driven by a signal that preferably includes at least one approximation of a step function. (Note that the step function is a mathematically ideal value that real-world circuits cannot achieve.) The step function includes a wide range of frequencies. By using a drive signal that includes an approximation of a step function, the electromechanical transducer is made to emit impulses of mechanical energy that propagate to a tactile point and are sensed by a user operating a mobile phone. In various embodiments, the electromechanical transducer is driven by a signal that includes one or more pulses. The pulses, e.g., a single pulse or a decoded waveform, are generated according to each detected state, where the state represents a specific situation identified by the phone processor. Using known pulses is advantageous in that known pulses generate impulses of mechanical energy that create a tactile sensation that simulates the feeling of a previous state that the user would be familiar with.

[0114] The transceiver module, telephone processor, A / D, input decoder, D / A 510, haptic interface module, display driver, memory, and display driver are preferably part of an electrical circuit embodied in the circuit components and interconnected to the wiring of the circuit board.

[0115] Alternatively, instead of using a telephone processor, a different electrical circuit could be used to drive an electromechanical transducer to generate haptic feedback at the touch point.

[0116] The haptic interface module could be a pulse generator that generates digital pulses of varying widths, heights, and / or frequencies based on instructions from the telephone processor. An amplifier may be required due to impedance matching to the electromechanical converter and the ability to source / sink current. Alternatively, the haptic interface module could simply be a current amplifier, with the pulses generated by the telephone processor itself. Another possibility is that the haptic interface module includes multiple DACs, which apply analog signals in case of additional audio channels.

[0117] Various situations can elicit different tactile responses. For example, on a pager or mobile phone, a message or call from a spouse might cause all tactile points to vibrate, or a message or call from a boss might cause tactile points to vibrate in a circular motion around the device, or a message or call from someone else might cause tactile points to vibrate repeatedly on one side of the device. The use of multiple consecutive adjacent vibrators, as described, creates the illusion of motion (known as a rabbit on the skin).

[0118] This illusion of movement can be used to convey directional information for movement. Movement along one side, around the electronic device, and back and forth can also be used to convey information such as attention-grabbing information, emphasis information, and general non-verbal information. The electronic device can also relay status information, such as out of range, low battery, and busy signals. Such information can be useful while the user holds the electronic device to their ear and cannot easily see the information on the screen.

[0119] Multiple localized force feedback can also be used for sensory communication. Instead of sending voice or text messages, or pictures or data files, specific haptic patterns can be sent to other users. The patterns may represent reminders, specific moods (e.g., I'm thinking of you, I love you, I miss you), specific emotions, or any other user-defined content.

[0120] Computer devices are widely used for entertainment activities such as playing games. Currently popular gaming computer devices include home television-connected game consoles such as the Nintendo 64 (Nintendo Corp.), PlayStation (Sony Corp.), and Dreamcast (Sega Corp.). Other gaming computer devices include personal computers such as Windows PCs and Macintosh computers. Portable computer devices, such as personal digital assistants like the Game Boy (Nintendo), PalmPilot (Palm Computing), and laptop computers, are also frequently used for entertainment purposes.

[0121] Users of these computer devices typically interact with games or other application programs using interface devices connected to a host computer (e.g., a game console). Such interface devices may include joysticks, gamepads, mice, trackballs, styluses, handles, or other devices. Users move user-operable objects (manipulators) such as joysticks, wheels, mice, buttons, dials, or other objects, which are sensed by the host computer and used to manipulate the graphical environment displayed by the host computer. Recently, haptic feedback has also become available in interface devices, where a microprocessor on the host computer and / or interface device controls one or more motors that output force to the user. These forces are associated with events or objects in the graphical environment to further immerse the user in the game experience or interface task. Here, the term “haptic feedback” is intended to include both tactile (or vibratory) feedback (forces transmitted to the user’s skin surface) and kinesthetic (forces provided in the degrees of freedom of the manipulator’s movement) feedback.

[0122] Current force-feedback "gamepad" controllers (or add-on hardware for gamepad controllers) used to interface with games running on game consoles include the DualShock® from Sony Corp., the Rumble Pak® from Nintendo Corp., and the Jump Pack from Sega Corp., as well as other types of handheld controllers such as the MadCatz Dual Force Racing Wheel. These devices are inertial haptic feedback controllers, which utilize one or more motors to vibrate the controller housing and thus provide a vibration-like output force to the user associated with game events and interactions. Typically, an eccentric rotating mass (ERM) motor, or pager motor, is used to generate vibrations on the controller and therefore to the user. The motor is rigidly connected to the controller housing and gives mass to the rotating shaft offset from the rotation of the axis, so that when the shaft rotates, the inertial force from the moving mass causes the motor and gamepad housing to oscillate back and forth.

[0123] To reproduce textures, force feedback devices are preferably used to allow the user to touch and feel computer-generated objects. The sensation of touch is preferably simulated using a haptic (sensory / touch) interface. A haptic interface is a force-reflecting device that allows the user to touch, feel, manipulate, create, and / or modify three-dimensional objects simulated in a virtual environment. A variety of known haptic interface objects exist, such as planar area interfaces, joysticks, gloves, thimbles, sticks or pens, exoskeleton structures, treadmills, fans, and magnets. Hardware that can be used includes DC brushless motors, potentiometers, Silicon Graphics, Inc. IRIS Indigo computers, V25 board computers, 8086-compatible microprocessors, CRT displays, stereoscopic imaging devices, magnetic and electromagnetic components, pulleys, steel belt-driven trains, VME buses, decoders, potentiometers, motor controllers, decoders, and cable reducers. The required software can be any of the various programming languages ​​(e.g., C, C++) that can work in conjunction with the visual modeling program.

[0124] Currently, there is no consensus among experts regarding the "best" type of interface. However, one known example of a haptic interface is the "Phantom Haptic Interface" developed at MIT's Artificial Intelligence Laboratory. The Phantom Haptic Interface delivers precise haptic stimuli to humans at a level that was previously unattainable, with greater fidelity and convenience. The device is designed to deliver force that generates "point contact," giving the sensation of fingertips interacting with a wide variety of objects. To achieve this, only three motors and three sensors are required, and the device provides a computationally and mechanically manageable method for enabling haptic interaction with complex virtual objects.

[0125] A haptic interface allows users to touch and interact with virtual, computer-generated objects in a way that evokes the sensation of "real" touch. This technology allows a user in front of a computer terminal to touch objects that exist only in the computer's "brain." By transmitting the correct digital signals to a master haptic interface device at the remote user's location, the master device can be used to make the user feel as if they are performing real-world tasks. In reality, the user may simply be interacting through a motor equipped with a computer program.

[0126] Various embodiments are optically based and commonly utilize inconspicuous special data that is on or embedded in an object whose 3D position and / or orientation is to be input into a computer. Typically, such data is seen by a single TV camera or two TV cameras forming a stereo pair. The camera's position may be near a computer display looking outward from it, or near a human work area or play area.

[0127] The counter-reflective glass bead tape, or beads such as Scotchlite 7615 from 3M Co., provides data of points, lines, or other desired shapes, which can be easily attached to any desired object and have high brightness and high contrast against surrounding objects such as people, cloth, rooms, etc., when illuminated by incident light along the optical axis of vision, such as the optical axis of a TV camera. This also allows the camera to have a fast integration time that can capture common motions used in normal environments and that are desired, and the data can be easily distinguished, which significantly reduces computer processing time and processing costs.

[0128] Figure 14a Figure 14a illustrates an exemplary camera based on an embodiment. In this case, in order to cause a display device C7 to generate a signal to activate an object or move it (for example, by subsequent finger motion or otherwise), user C5 wants to point to an electronically represented object C6 on the screen C7 and register the pointing motion with respect to the object (virtual object) in software contained in computer C8. He achieves this using a TV camera C10 positioned above the screen, as typically shown, or positioned to the side (as in C11), to determine the position of his fingertip C12 in space and / or the direction of his finger's pointing C13.

[0129] In some cases, it may be desirable to use antireflective materials on the finger (for example, temporarily attached to the finger as jewelry, painted on the finger using antireflective coating "nail polish," or attached to the finger with adhesive tape having an antireflective coating). Such coatings include Scotch-lite 7615 and its equivalents, which have a specific reflectivity that makes them easily identifiable and provide good contrast to their surroundings. The brightness of the reflection enables dynamic target acquisition and tracking at the lowest cost.

[0130] The use of back-reflecting and / or highly distinctive targets (e.g., a glowing orange triangle) enables reliable target acquisition in common situations and does not limit the device to pointing on desktop applications under controlled lighting. Active (self-illuminating) targets such as LEDs can also enable such acquisition.

[0131] If we consider a camera system C10 located above screen C7 and viewing the user, more specifically the user's hands, then in the typical case of an internet phone, it has a relatively large field of view, large enough to also view the user's face. This same field of view can be used in various embodiments, but it describes a relatively large quantity. For higher precision, add-on lenses or zoom lenses on the camera can be used to increase the resolution.

[0132] Alternatively, in various embodiments, it is possible to have multiple cameras, one for the internet and others for the input applications described herein. Indeed, with prices constantly falling, the cost of actual cameras with plastic lenses on CMOS chips is very low, and it is probably possible to have a number of fixed-magnification cameras, each with a separate chip!

[0133] These can be easily daisy-chained via either FireWire or USB, and as a result, they can be electronically selected either by different magnifications or by pointing in the desired direction.

[0134] Returning to the problem of determining the position or orientation of a human body part, typically a hand or finger, and in this case a finger. In various embodiments, low-cost lighting can be used. Power for lighting, such as LEDs, can generally be supplied in any way via USB or a 1394 bus.

[0135] The user can also point or signal with an object such as C15 having data C16, such as a reverse reflection point C16 or a line target C17, on top of it.

[0136] The sensing of the 2D position described above can be extended to 3, 4, 5, and 6 dimensions (x, y, +z, pitch, yaw, roll). Two of the many possible sensing capabilities are described herein in various embodiments. 1. The first possibility, illustrated in Figures 14a and 14b, is to utilize a single camera but to leverage numerous distinct features or other targets on an object that can provide multi-degree-of-freedom solutions. In one example, the target spacing on the object can be predetermined and entered into the computer manually or automatically from software containing data about the object, or determined through a taught decision process. 2. A second possibility is a solution using two cameras, as shown in Figures 14c and 14d, which does not require prior knowledge of the target and can actually determine the 3D position of a single target on its own, which is useful for determining the position of, for example, a fingertip. With six degrees of freedom of information, even for a line target, the target requires at least three points, and combinations of lines and points can also be used.

[0137] Figure 14b illustrates an embodiment of 3-D (three-dimensional) sensing that utilizes a single stereo camera with three or more data points on the object to be sensed, or in another example, on the user's wrist.

[0138] As shown, the user has an object C30 in his right hand C29, which has at least three visible data C32, C33, and C34, which are seen by a TV camera C40, the signals of which are processed by a computer C41, which also controls a projection display device C42. The TV camera C40 also sees three other data C45, C46, ​​and C47 on the user's left wrist C48 to determine the direction and approximate orientation of the pointing finger of the left hand C51, or its orientation relative to the object C30, or any other data, or any other data (e.g., data relative to the screen position, or other position relative to the mounting position of the TV camera), or the user's head if seen, or anything else. The position and orientation of the object and hand can be determined from three points in the camera image using known photogrammetry equations (see Pinckney, citation U.S. Patent No. 4219847 and other references in the cited literature).

[0139] Alternatively, for three distinct point targets, a colored triangular target can be used, for example, where the intersection point of the lines fitted on both sides defines the target data as described below.

[0140] Similarly, camera C40 can be used to view objects of other purposes. The direction in which the user object C55, displayed on the display device C42, points is determined, for example, by data C50 on the finger C52 of the user's left hand C51 (the position and tilt of the wrist can also be determined).

[0141] Alternatively, the finger can also be detected from its general grayscale image alone and can be easily identified relative to the target wrist position (especially if, as shown, the user is clenching his other fingers, resulting in only finger C52 being extended in that hand).

[0142] The computer can process grayscale images using known techniques, such as other algorithms found on the Matrox brand Genesis image processing board for blobs and PCs, and then use the wrist knowledge derived from the data to determine the direction of finger pointing. This allows the fingers C50 of the left hand to alternatively point to (or touch) a point to be determined on an object C30 held in the right hand.

[0143] Figure 14c Figure 14c illustrates another version of the embodiments of Figures 14a and 14b, in which two cameras, “binocular” stereo cameras C60 and C61, processed by a computer C64, are used to image C65, an artificial target (in this case a triangle; see also Figure 2) at the end of a pencil C66, and optionally to improve the pointing resolution of a target C67 at the end of the pencil, typically at a known small distance from the tip (for clarity, the user and the user’s hand holding the pencil are not shown). This imaging allows tracking the tip position of the pencil to determine where the pencil is making contact on the paper (or TV screen in the case of a touchscreen).

[0144] To provide independent illumination of the back-reflected target for each camera, it may be desirable to have nearly coaxial light sources C62 and C632, controlled independently by the computer C64 shown. This is because the back-reflector reflects differently at different approach angles, and since the cameras are often angularly spaced (e.g., by an angle A that is not zero), they will not be looking at the same target.

[0145] Numerous other camera configurations, processing, calculations, and other issues are generally discussed in relation to the precise determination of object positions using stereoscopic systems of two or more cameras in the paper by SFEI Hakim referenced above and other references mentioned therein.

[0146] The computer can also acquire a three-dimensional image of the paper and the target at the four corners C71-C74. The solution to the photogrammetry equations allows the position of the paper relative to the camera in space to be determined, and therefore the position of the pencil relative to the paper, in particular the position of its tip, to be determined, which is communicated to the display means C75 or other computer program. Even if there is no target at the edge, the pointing direction can be determined by knowing the length of the target C65 and the calculated position of the pencil tip.

[0147] Line target C76 may be useful on a pencil, or multiple line targets spaced circumferentially may also be useful for defining the direction the pencil is pointing from a pair of stereoscopic images.

[0148] The range of motion of the measurement system is shown by the dotted line C79, which in this case is the area on and above the desktop where the sensor system can operate effectively. Typically, this is sufficient for the task at hand. Note that, due to possible camera decoding tilts and other geometric considerations, the effective range of motion for any accuracy or resolution criterion does not necessarily have parallel sides.

[0149] It should be noted that the two (stereo pair) camera systems in Figure 14 have been widely tested and can provide extremely accurate positional and directional information up to six degrees of freedom. One particular version, using a commercially available CCD monochrome camera, a Matrox "Genesis" frame grabber board, an image processing board, and suitable stereophotogrammetry software running on an Intel Pentium 300MHz-based computer, has features that make it well-suited, for example, to input from a large desktop CAD station. This provides 30Hz updates of six-axis (x, y, z roll, pitch, and yaw) data over a movable range of 0.5m x 0.5m in x and y (desktop with the camera directly overhead pointing to the desk) and 0.35m in z on the desk, all with an accuracy of 0.1mm or better, for example, when using data of a clearly identifiable round back-reflected object (scotchlite 7615-based) with a diameter of approximately 5-15mm on an object. This can be accurate enough for precise tasks such as designing objects in a 3D CAD system.

[0150] In this example, the camera is mounted overhead. If it were mounted to the side, in front, or at an angle such as 45 degrees to the desktop, the z-axis would be outward from the camera.

[0151] Figure 14c further illustrates the three-dimensional arrangement of two cameras used in this case to determine the position and orientation of an object with a line target and data over a portion of the user. Here, cameras C60 and C61 are positioned to view a reverse-reflected line target C80 that runs along a portion of the length of the toy sword blade C81. The line target in this case is created as part of the plastic sword and has the shape of a box-shaped reflector molded into the corner, similar to the reflectors in a car's taillight reflector. It may also be created to have a distinctive color relative to the rest of the sword, and the combination of the two provides an unmistakable indication.

[0152] Typically, when viewed in a back-reflected manner, no other bright lines are present in any typical image. This also illustrates how the shape of a target (e.g., a line) can be used to identify other unwanted glare and reflections that may contain some valuable bright pixels in an image. Note that if a line-type target is wrapped around a cylindrical object, its shape can be cylindrical, and as a result, it can be viewed from multiple angles.

[0153] The alignment of the two camera images and the solution to the photogrammetry equations give the pointing direction of the line target. If an additional point such as C82 is used, a complete 6-degree-of-freedom solution for the sword is available. Also shown here is yet another point C83, which serves two purposes: it enables an improved photogrammetry solution and it serves as a surplus target when C82 is not visible due to ambiguity, obscuration, or other reasons.

[0154] This data is calculated in the C64 computer and used to modify the display as desired.

[0155] In one embodiment, a Matrox Genesis frame processor card on an IBM 300MHz PC was used to read both cameras and process the information at a camera frame rate of 30Hz. Such line targets are very useful for outlining the edges or parts of an object, such as the sleeves of clothing, the seams of pointing gloves, the brims of hats, and other decorative and practical purposes, such as holes or tears.

[0156] Typically, cameras C60 and C61 have equal magnification and field of view, and the desired measurements overlap. The camera axes can be parallel, but for operations within a range of less than 2-3 meters, they are tilted at an acute angle A to increase the overlap of their fields of view, especially when a large baseline distance d is used to improve accuracy (although the z range is less likely). For example, for CAD drafting applications, with a baseline of 0.5-1 meter, A could be 30-45 degrees. In the case of video games, as shown in Figure 5, the z range can be 5 meters or more, and the angle A and baseline become smaller, allowing for a larger range of activity.

[0157] database Data on an object may be known in relation to other aspects and other data of the object by selling or otherwise providing to a user an object designed with such knowledge, and by including in it a CD-ROM disk or other computer-interfaceable storage medium containing this data. Alternatively, a user or someone can impart this information to a computer system. This is particularly useful when the data is applied by a user to any object.

[0158] Figure 14d Illustrated here is a process used in various embodiments relating to the detection of a single point for creating a command, in which the position (or change in position, i.e., movement) of a fingertip having a mounted back-reflective target is detected by a pair of stereo TV cameras using a detection algorithm based on image thresholding in the simplest case, so that only the bright target indication is seen from the finger (and optionally, any associated object such as a screen to be touched).

[0159] If this is insufficient to define the data on the finger without ambiguity, additional algorithms known in the field can be used (many of which are commonly found on image analysis frame grabber boards such as Matrox Genesis). This processing may include: A process of detecting brightness by comparing it to the surroundings or very close areas (contrast); A shape detection process in which shapes such as circles, rings, and triangles are searched for; A color detection process in which a specific color is searched for; A movement process in which only target candidates that have moved from their position in the previous TV image are visible.

[0160] Each process may only process items that have passed the previous process, or each may be executed independently, with the results being compared later. The order of these processes can be changed, but any changes are made to further identify appropriate finger targeting.

[0161] Next, the position of the targeted finger is determined by comparing the position of the finger target in the two camera images of the stereo pair. In this case, since a single target is used, there is no alignment problem, and it only occurs when a point is found in each image.

[0162] After an image of the fingertip (or other tool) is found, its position is calculated relative to the screen or paper, and this data is input into a computer that controls the display, for example, to determine the position of a drawing line, change an icon, or determine the vector of movement on the screen.

[0163] Motion detection Computer 8 can be used to analyze the incoming TV image-based signal to determine which points in the image are moving. This is useful for removing stationary background data, as the target is often only moving items, such as hands or objects. Furthermore, the direction of movement is often the desired answer, or the fact that movement occurs everywhere.

[0164] A simple way to determine this is to subtract the image of a high-contrast back-reflected target from the first image, and then determine which parts are different, essentially representing the movement of the points. Small changes in lighting or other effects are not registered. Similarly, obviously more sophisticated algorithms exist.

[0165] Motion preprocessing is useful when the target contrast is not very high, as it removes irrelevant areas and focuses all target recognition and measurement processing on the actual target item.

[0166] This type of processing is also useful when two-camera stereo is used, as only moving points are considered in image matching; the problem arises when there are many points in the field.

[0167] Can an object be considered to be moving? The answer is "yes" if it's a game or many other activities. However, there can be issues with the speed of movement. Perhaps in games, the frame rate is the standard, i.e., 30Hz for a typical camera. However, in some cases, movement can be defined as slower, for example, 3Hz for CAD system inputs using the designer's careful motion.

[0168] Once movement data is identified, its range can then be determined, and if the object is subsequently tracked, even if it does not move forward from that point, measuring the range provides a good way to fixate on objects using more than just two dimensions.

[0169] Humans can actually use artificial movement of a target if it does not exist in nature. This can be done by making it vibrate. If one or more LEDs are used as a target, they can be made to blink, and this also appears in image subtraction (image with LEDs versus image without LEDs). The same applies to targets with altered colors, and this also appears in color image subtraction.

[0170] Image subtraction or other computer processing operations can also be useful in other senses. It is possible to subtract the background, activating the back-reflected illumination light where there is no back-reflected target, and then subtracting them. Another idea is to simply take a photograph of a room or other workspace and then capture the targeted object. Subtraction or something like that seems very simple. The final result is that any irrelevant, luminous features in the space, such as shiny doorknobs, glass, etc., are removed from consideration.

[0171] This can also be done with colored targets using color-based image subtraction, which is particularly useful when the desired color is known in advance (through instruction mode).

[0172] The flowchart shown in Figure 14d illustrates the following steps: A. Obtain images of a stereo pair; B. Optionally, preprocess the image to determine whether motion is present. If it exists, move to the next step; otherwise, do not move to the next step or may move (as desired); C. Threshold the image; D. If the brightness is insufficient, change the brightness acquisition parameters such as brightness or integration time; E. Identify the target; F. If identification is not possible, add other steps, such as screening for the target's color, shape, or size; G. Determine the centroid or other features of the target point (in this case, the inverse reflection point on the finger); H. If necessary, perform auxiliary alignment processes; I. Compare the positions in the stereo pair to determine the range z, x, and y of the target position; J. An auxiliary step for determining the position of a target on a screen when the screen's position is unknown to the computer program. This involves determining what is projected onto the screen, for example, a housing, via the target on the screen; K. Determine the target's position relative to the screen; L. Determine the point in the specified display program; M. Modify the display and program as desired.

[0173] Figure 14e The following describes multi-degree-of-freedom image processing of a triangular-shaped color target (disclosed in some embodiments herein), which can be optically found using one or more cameras to obtain the three-dimensional position and orientation of the target using the computer-based methods described below. Advantageously, it is best suited for targets defined by a large number of pixels in the image plane, as well as for color processing, as well as for the highest resolution, typically because the target is large, or the camera is close to the target, or the camera's field of view consists of a very large number of pixels.

[0174] The method is simple, but unique in that it can be applied to 1) increase accuracy to varying degrees (at the expense of speed), 2) use one or more cameras (more cameras increase accuracy), and 3) utilize the target color and combinations of (one or more) triangles to identify a tool or object. It utilizes the edges of triangles to obtain precise sub-pixel accuracy. The method can work even better if the edges of the triangles have gentle curves. Other geometric shapes can sometimes be handled similarly.

[0175] The method is based on precisely defining the edges and then calculating the intersection points of the curves of these edges to find the three vertices (F0, G0, F1, G1, F2, G2) of each triangle in the camera's field of view. This is generally more accurate than finding three or four points from the centroid region. However, the choice of which to use often boils down to the question of which is more satisfactory to the consumer or more durable and reliable in use.

[0176] In a preferred embodiment, one or more cameras are used to capture a target object consisting of brightly colored right triangles on a rectangle of a background material of different bright colors. The background color and the triangle color must be two colors that are easily distinguishable from the rest of the image. For illustrative purposes, the inventors will describe the background color as bright orange and the triangle as light blue.

[0177] By using the difference between the background color and the triangle's color, the vertices of this triangle can be found with great accuracy. If there are two or more triangles above the target object, accuracy can be further improved by using a weighted average of the positional and orientation information.

[0178] This method begins by searching for pixels that have the background color or the triangle's color, starting from the position of the centroid pixel of the triangle from the previous frame. Once a pixel with the triangle's "light blue" color is found, the program moves in four opposing directions, continuing until each advance detects a color indicating an edge that divides the triangle and the "orange" background. The method then extends this edge and defines the three sides of the triangle using the least squares method. The intersection of the resulting three lines is found, which serves as a rough estimate of the triangle's vertices. These can serve as input for applications that do not require high precision.

[0179] If greater precision is desired, these provisional lines are then used as a starting point for the sub-pixel refinement process. Each of these three lines is checked to determine if it is mostly horizontal. If the line is mostly horizontal, the new line is determined by fitting the best fit of the curve across the pixels in each column that straddles the provisional line. If the line is mostly vertical, the same process proceeds across the pixels in the horizontal column.

[0180] The color of each pixel that the line crosses is translated into a corresponding numerical value. A pixel that is purely cyan will be given a value of 0, while a pixel that is purely orange will be given a value of 1. All other colors generate a number between 0 and 1 based on their relative amounts of cyan and orange. This numerical value, V, assigned to a pixel is the weighted average of its color components (R, G, B values, etc.). If the calibrated cyan components are AR, AG, AB, the orange components are OR, OG, OB, and the pixel components are PR, PG, PB, then the numerical value V is: V = WR*CR + WG*CG + WB*CB In the formula, WR, WG, and WB are weighting constants between 0 and 1, and CR is defined as follows: The same process can be used to define CG and CB.

[0181] This value V is compared to an ideal value U, which is equal to the percentage of orange calculated assuming the angle of the provisional line is the same as the angle of the ideal line. For example, a pixel that is precisely crossed in the middle by that line would have a U of 0.5, since it is 50% cyan and 50% orange. Fitting UVs in a column (or row) near the provisional line crossings gives a new estimate of the location of the true edge crossings. Finally, these sets of crossings can be fitted to lines or gentle curves for each of the three edges, and the three vertices can be calculated from the intersections of these lines or curves.

[0182] These three precise vertices can be used in the camera plane (F0, G0, F1, G1, F2, G2) along with the lens formula (in this specification, for simplicity, the inventors will simply use the lens formula) to relate the x and y of the target object to F and G. F=λX / Z; G=λY / Z λ is the focal length, and z is the vertical distance from the lens to the target object's position. The triangle on the target object is initially defined as being on a plane parallel to the lens plane. In a preferred configuration, there is a right triangle with a right angle defined by x0, y0, z0, having one side (of length A) extending along the camera's F-axis and the other side (of length B) extending along the camera's G-axis. The orientation of the real target object is associated with this orientation using Euler angles φ, θ, ψ. Along with the lens equation and the Euler equation, the six derived data values ​​of the three vertices (F0, G0, F1, G1, F2, G2) can be used to define six values ​​for the target object's position and orientation. The position and orientation of a point of interest on any tool or object rigidly attached to this target object can be easily calculated from calibration data as well as the usual translation and rotation transformations. Lens distortion can be addressed by forming a correction function using calibration data that corrects the positions of the F and G data. Euler's equation is nonlinear. The inventors linearized Euler's equation by first assuming that these angles have not changed much since the last image frame. Thus, φ is replaced with φ(old) + U1, θ is replaced with θ(old) + U2, ψ is replaced with ψ(old) + U3, and z0 is replaced with z0(old) + U4, i.e.: φ = φ + U1 θ = θ + U² ψ = ψ + U³ z0 = z0 + U4 This can be done. Substituting these into Euler's equation and applying the lens formula, we get the matrix equation SU=R This is obtained. This can be solved for the U-value using standard methods such as the Gauss-Jordan routine. The angle and z0 can be updated iteratively until convergence is obtained. The coefficients of this matrix are: s11=-A(cos(φ)(F1 / λcos(ψ)+sin(ψ))-sin(φ)cos(θ)(F1 / λsin(ψ)-cos(ψ))) s12=Asin(θ)cos(φ)(F1 / λsin(ψ)-cos(ψ) s13=A(sin(φ)(F1 / λsin(ψ)-cos(ψ))-cos(φ)cos(-θ)(F1 / λcos(ψ)-sin(ψ))) s14=(F0-F1) / λ s21=A(G1 / λ(-cos(φ)*cos(ψ)+sin(φ)sin(ψ)cos(θ))+sin(θ)sin(φ)) s22=Acos(φ)(G1 / λsin(θ)sin(ψ)-cos(θ)) s23=G1 / λA(sin(ψ)sin(φ)-cos(ψ)cos(θ)cos(φ)) s24=(G0-G1) / λ s31=0 s32=-Bcos(θ)(F2 / λsin(ψ)-cos(ψ)) s33=-Bsin(θ)(F2 / λcos(ψ)+sin(ψ)) s34=(F0-F2) / λ s41=0 s42=-B(G2 / λsin(ψ)cos(θ)+sin(θ)) s43=-BG2 / λsin(θ)cos(ψ) s44=(G0-G2) / λ Defined as r1=(F1-F0)z0 / λ+A(F1 / λ(cos(ψ)sin(φ)+cos(θ)cos(φ)sin(ψ))+sin(ψ)sin(ψ)-cos(θ)cos(φ)cos(-ψ)) r2=(G1-G0)z0 / λ+A(G1 / λ(cos(ψ)sin(φ)+cos(θ)cos(φ)sin(ψ))+sin(θ)cos(φ)) r3=(F2-F0)z0 / λ+Bsin(θ)(F2 / λsin(ψ)-cos(ψ)) r4=(G2-G0)z0 / λ+B(G2 / λsin(θ)sin(ψ)-cos(θ)) defined as follows. After the rest of the parameters x0 and y0, the equation: x0=F0z0 / λ Y0 = G0z0 / λ is defined from

[0183] Conspicuous color transitions can provide much more information than black-and-white transitions and are useful for accurately calculating the position and orientation of an object. As color cameras and high-capacity processors become less expensive, the additional information provided can be accessed virtually at no additional cost. And very importantly, in many cases, color transitions are more comfortable for the user to view than plain black and white. Additionally, the color can vary within the target object to create further opportunities for statistically enhancing the resolution at which the target object can be found.

[0184] (Problems in 3D Input to a Computer) Today, inputting 3D information into a computer is often laboriously done using 2D devices such as a mouse or similar device. This approach is unnatural for the person, as well as for the program and the interaction with that person, and CAD designers working with 3D design systems require many years of experience to acquire the skills necessary to design efficiently using this approach.

[0185] A similar situation exists for very common computer video games where the content is much more three-dimensional and graphically image-like, but with similar limitations. These games have also not been natural for players (one or more) so far.

[0186] "Virtual Reality" also requires 3D input for head tracking, movement of body parts, etc. This has led to the development of sensor capabilities in additional areas that have provided some solutions, but these are either cumbersome for the user, expensive, or both.

[0187] The limitations of 3D computer input have also restricted its use in naturalistic settings such as medical lectures and simulations. Furthermore, it limits the benefits that infants, the elderly, and people with disabilities can derive from computer-assisted living and work.

[0188] Another aspect is the digitization of object shapes. Sometimes, as a starting point for 3D design, you want to have a plastic model or a real-world part in your hands.

[0189] The inventors propose a single, inexpensive device that can provide all of these controls and can serve as a drafting pad, input 3D-modeled shapes, or even allow the user to use actual clay, with the computer recording the new shape as the user sculpts with the clay.

[0190] Various embodiments associate physical activity and parts of the body with computer commands. A novice user can design a house using a set of purposeful models or "toys" such as doors, windows, and walls. By touching the appropriate toy elements and then moving or rotating the user's hand, the user can position those elements correctly. The user can obtain visual stimulation either by viewing the toy's position on a desk or by viewing a scaled representation on a computer display. Many other embodiments are also possible.

[0191] (Object tracking) In one general embodiment, a method for tracking an object of interest is disclosed. This method includes the steps of acquiring a first image and a second image representing different viewpoints of the object of interest, and processing the first image into a first image dataset and the second image into a second image dataset. The method further includes processing the first and second image datasets to generate a background dataset related to the background, and generating a first difference map by measuring the difference between the first image dataset and the background dataset, and a second difference map by measuring the difference between the second image dataset and the background dataset. The method also includes the steps of detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map, and generating an absolute position of the object of interest from these first and second relative positions.

[0192] The process of processing the first image into a first image dataset and the second image into a second image dataset may include the steps of determining the active image region for each of the first and second images, and extracting the active image dataset from the first and second images contained within this active image region. The step of extracting the active image dataset may include one or more of the following methods: cropping the first and second images, rotating the first and second images, or shifting the first and second images.

[0193] In one implementation, the step of extracting valid image datasets may include the step of arranging these valid image datasets into image pixel sequences having rows and columns. The extraction step may further include the step of identifying the maximum pixel value in each column of the image pixel sequence, and the step of generating a dataset having one row, wherein the identified maximum pixel value for each column represents that column.

[0194] The process of processing a first image into a first image dataset and a second image into a second image dataset may also include a step of filtering the first and second images. The filtering step may include a step of extracting edges from the first and second images. The filtering step may further include a step of processing the first and second image datasets to highlight the differences between the first image dataset and the background dataset, and to highlight the differences between the second image dataset and the background dataset.

[0195] Processing first image data and second image data to generate background datasets may include generating a first set of one or more background datasets associated with the first image dataset, and generating a second set of one or more background datasets associated with the second image dataset.

[0196] Generating a first set of one or more background datasets may include generating a first background set representing the maximum value of data in a first image dataset representing the background, and generating a second set of one or more background datasets may include generating a second background set representing the maximum value of data in a second image dataset representing the background. Generating may further include increasing the values ​​contained in the first and second background sets, set by a predetermined value, for the first and second background sets representing the maximum value of data representing the background.

[0197] Generating a first set of one or more background datasets may include generating a first background set representing the maximum value of data in a first image dataset representing the background, and generating a second set of one or more background datasets may include generating a second background set representing the minimum value of data in a second image dataset representing the background. Generating may further include reducing the values ​​contained in the first and second background sets, set by a predetermined value, for the first and second background sets representing the minimum value of data representing the background.

[0198] Generating a first set of background datasets may include sampling a first image dataset, and generating a second set of background datasets may include sampling a second image dataset. Sampling may occur automatically at predetermined time intervals, where each sample may include data unrelated to the background.

[0199] Generating a first set of one or more background datasets may include maintaining multiple samples of a first image dataset within each background dataset, and generating a second set of one or more background datasets may include maintaining multiple samples of a second image data set within each background dataset.

[0200] Generating each first background dataset may include selecting, from a plurality of samples, one value representing the background for each element within the first image dataset, and generating each second background dataset may include selecting, from a plurality of samples, one value representing the background for each element within the second image dataset. Selecting may include selecting the median value from all sample values in each of the background datasets.

[0201] In other implementations, generating may include comparing the first image dataset with a subset of the background dataset, and comparing the second image dataset with a subset of the background dataset.

[0202] In other implementations, generating the first difference map may further include representing, as one of two states, each element within the first image dataset, and generating the second difference map may further include representing, as one of two states, each element within the second image dataset, where the two states represent whether the value is consistent with the background.

[0203] In yet another implementation, detecting may include identifying clusters in each of the first and second difference maps, where each cluster has elements and the state of the elements in the associated difference map indicates that the element is inconsistent with the background.

[0204] Identifying clusters may further include reducing the difference map to one row by counting the elements within columns that are inconsistent with the background. Identifying clusters may further include identifying columns as being within a cluster and classifying adjacent columns as being within a cluster. Identifying columns as being within a cluster may also include identifying the central column.

[0205] Identifying clusters may further include identifying locations associated with those clusters. Identifying locations associated with clusters may include calculating a weighted average of the elements within the clusters.

[0206] Detection may further include classifying the cluster as the target. Classifying the cluster may further include counting the elements within the cluster and classifying the cluster as the target only if the count exceeds a predetermined threshold. Classifying the cluster may further include counting the elements within the cluster, counting the total number of elements classified as inconsistent in the background of the difference map, and classifying the cluster as the target only if the ratio of the count of elements within the cluster to the total number of elements exceeds a predetermined threshold.

[0207] The detection process may further include identifying subclusters within the cluster representing the pointing end of the target, and identifying a portion of the subclusters.

[0208] In the implementation described above, the object may be the user's hand, and the method may include controlling an application program that uses an absolute portion of the object.

[0209] The implementation described above may further include obtaining a third and a fourth image representing different viewpoints of the object, processing the third image into a third image dataset and the fourth image into a fourth image dataset, and processing the third and fourth image datasets to generate a background dataset related to the background. The method may also include generating a third difference map by determining the difference between the third image dataset and the background dataset, generating a fourth difference map by determining the difference between the fourth image dataset and the background dataset, and detecting the third relative position of the object in the third difference map and the fourth relative position of the object in the fourth difference map. The absolute position of the object may be generated from the first, second, third, and fourth relative positions of the object.

[0210] As part of this implementation, the object may be the user's hand, or it may include controlling an application program that uses the absolute location of the object.

[0211] In another embodiment, a method is disclosed for tracking an object controlled by a user coupled with a computer. The method includes acquiring images from at least two viewpoints, processing the acquired images to generate an image dataset for each acquired image, and comparing each image dataset with one or more background datasets to generate a difference map for each acquired image. The method also includes detecting the relative position of the object within each difference map, generating the absolute position of the object from its relative position, and using the absolute position so that the user can interact with a computational application.

[0212] Furthermore, this method may include mapping the absolute position of the object to screen coordinates associated with a computer application, and using the mapped position to connect with the computer application. This method may also include recognizing gestures associated with the object by analyzing changes in the absolute position of the object, and combining the absolute position and gestures to connect with a computer application.

[0213] In another embodiment, a multi-camera tracking system is disclosed that interacts with an application program running on a computer. The multi-camera tracking system includes two or more video cameras configured to provide different viewpoints of a region and are operable to generate a series of video images. A processor is operable to receive the series of video images and detect objects appearing in the region. The processor generates a background dataset from the video images, generates an image dataset for each received video image, compares each image dataset to the background dataset to generate a difference map for each image dataset, detects the relative position of the object in each difference map, generates the absolute position of the object from its relative position, and maps the absolute position to a position indicator associated with the application program.

[0214] In the implementation described above, the object may be a human hand. Furthermore, the area may be defined in front of a computer-related video display. The processor can be operated to map the absolute position of the object to a position indicator such that the position indicator on the video display is aligned with the object.

[0215] The region may be defined at any distance in front of a computer-related video display, and the processor may be operable to map the absolute position of an object to the position indicator on the video display such that the position indicator on the video display is aligned with the position pointed to by the object. Alternatively, the region may be defined at any distance in front of a computer-related video display, and the processor may be operable to map the absolute position of an object to the position indicator on the video display such that the movement of the object is estimated as a larger movement of the position indicator on the video display.

[0216] The processor may be configured to emulate the functionality of a computer mouse. This may include configuring the processor to emulate controlling computer mouse buttons using gestures derived from the movement of the object. The sustained position of the object for a predetermined period of time may trigger a selection action within the application program.

[0217] The processor may be configured to emulate controlling computer mouse buttons based on the sustained position of the object for a predetermined period of time. Sustaining the position of the object within the interactive display area for a predetermined period of time may trigger a selection action within an application program.

[0218] The processor may be configured to emulate controlling computer mouse buttons based on the sustained position of a position indicator within an interactive display area for a predetermined period of time.

[0219] In the embodiments described above, the background dataset may include data points representing at least a portion of a static structure. In this embodiment, at least a portion of the static structure may include a patterned surface that is visible to the video camera. This static structure may be a window frame, or it may include a fragment of light.

[0220] In another embodiment, a multi-camera tracking system for interacting with an application program running on a computer is disclosed. The system includes two video cameras configured to provide different viewpoints of a region and is operable to generate a series of video cameras. A processor is operable to receive a series of video images and detect objects appearing in the region. The processor generates a background dataset from the video images, generates an image dataset for each received video image, compares each image dataset to the background dataset to generate a difference map for each image dataset, detects the relative position of the object in each difference map, generates the absolute position of the object from its relative position, identifies the subregion indicated by the object, and, if the object occupies an identified subregion, associates an action with the identified subregion being invoked and performs processing to apply that action to interact with the application program.

[0221] In the implementation described above, the object may be a human hand. Furthermore, the action associated with the identified sub-region may emulate the activation of a keyboard key associated with an application program. In the implementation described above, maintaining the position of the object in any sub-region for a predetermined period of time may trigger the action.

[0222] Details of one or more implementations are described in the attached drawings and the following description.

[0223] Figure 15 shows a multi-camera motion tracking and control system D100 interacting with an image viewing system. In this implementation, two cameras D101 and D102 scan the region D103. A controlled or known background D104 surrounds region D103. Region D105 is tracked by the system when it enters region D103. The object D105 is any common object inserted into region D103, typically the hand or finger of a system user. The object D105 may also be a selection device such as a pointer.

[0224] A series of video images acquired from cameras D101 and D102 are transmitted to a computing device or image processor D106. In this implementation, the computing device is a general-purpose computer running additional software that provides feedback to the user on a video display D107.

[0225] Figure 16A shows a typical implementation of the multi-camera control system D100. Two cameras D101 and D102 are positioned outside the region D103. The cameras are oriented so that the intersection D204 of their fields of view (D205 for camera D101, D206 for camera D102) encompasses the region D103. The orientation is such that cameras D101 and D102 rotate on an axis that is substantially parallel to each other. In this example, floor or window protrusions and side walls provide a controlled background D104 with prominent edges. The corresponding field of view captured by camera D101 is shown in Figure 16B. Although not shown in the figure, the field of view captured by camera D102 is a symmetrical image to the field of view captured by camera D101. The controlled background D104 does not have to cover the entire field of view D205 of the cameras. For each camera, the active image region D208 is entirely contained within the controlled background D104, and also contains the entire region D103. The background D104 has modelable features, and the object D105 is controlled so that its features differ from those of the background D104 in part or in whole. When the object D105 appears within region D103, the object D105 obstructs a portion of the controlled background D104 within the active image region D208 of each camera D101 and D102. At the location of the obstruction, the obstructed image, either in whole or in part, is inconsistent with the model of the controlled background D104 with respect to the selected features.

[0226] In summary, if the target D105 is identified and found, its position within the active image region D208 of both cameras is calculated. Using the position data of cameras D101 and D102, the camera positions associated with region D103, and parameters describing the cameras, the position of the target D105 within region D103 is calculated.

[0227] The processing performed by the image processor D106 (Figure 15), which may be carried out via software or hardware, is schematically shown in Figure 17. Camera images are simultaneously transported from cameras D101 and D102 and transferred to image buffers D306 and D307 (each) in the image processor D106, which are captured by image acquisition modules D304 and D305 (each). Image detection modules D308 and D309 independently detect the target D105 in each image and determine its position relative to the camera field of view. Relative position information D310 and D311 from both camera fields of view are combined by the combination module D312 and, if necessary, fine-tuned by the position fine-tuning module D313 to determine the global presence and the position of the target D105 within the region D103 in block D314. If necessary, specific gestures performed by the user may be detected by the gesture detection module D315. The results of the gesture detection process are then carried to another process or application D316, either on the same image processor D106 or on a different processing device. The gesture detection process is described in more detail below.

[0228] Image detection modules D308 and D309 are identical in the processing they perform. Implementations of these image detection modules D308 and D309 are shown in Figure 18. In block D402, image processor D106 extracts image data corresponding to the active image region D208 (in Figure 16B) from the captured image data stored in image buffer D306 or D307. The image may be filtered in filtering process D403 to highlight or extract image features or characteristics, where the background D104 and the target D105 are different, but otherwise remain constant within background D104 over time. In some implementations, the data representing the active image region may also be reduced by scaling module D104 to reduce the computational load required in later processing steps. Using the resulting data, background D104 generates one or more descriptions, which are modeled by one or more instances of background model processing in block D405 and represented as background model data 406 of the controlled background D104. Therefore, the background D104 is modeled with respect to a desired aspect or feature of the image. The background model D406 is transformed into a set of criteria in processing D407. In the comparison process D408, the filtered (from processing D403) and / or reduced (from module D404) image data is compared to these criteria (from processing D407), and the locations where the current data is inconsistent with the background model data D406, i.e., locations where the criteria are not met, are stored in the image or difference map D409. In the detection module D410, the difference map D409 is analyzed to determine whether any such inconsistency is suitable as a possible indication of the subject D105 and whether these criteria are met, and its location within the camera field of view (D205 or D206) is determined.The position of object 105 may be further fine-tuned in block D411 (if necessary) to generate a camera-related presence and a position output D310 or D311 related to object D105 (as described above in relation to Figure 17).

[0229] In block D402 of Figure 18, the image processor D106 extracts image data corresponding to the active image region D208 (in Figure 16B). The image data may be extracted by cropping, shearing, rotating, or transforming the captured image data. Cropping extracts only a portion of the entire image within the active image region D208. A bound is defined, and any pixels within the bound are copied, unmodified, and sent to a new buffer, while pixels outside the bound are ignored. The active image region D208 may be of any shape. Shearing and rotation rearrange the data into a more convenient order for further processing, for example, into a rectangle, and as a result, the pixels can be addressed in units of rows and columns.

[0230] Rotation makes the content of an image appear as if it is rotating. Rotation rearranges the positions from (x,y) to (x',y') according to the following equation: × (times) × θ × × θ × × θ × × θ × × θ, where θ is the angle at which the image is rotated.

[0231] When cameras D101 and D102 are properly mounted to region D103, the desired angle of rotation is usually small. When the desired angle of rotation is small, shearing may be used to provide a simpler approximation than rotation for computational purposes. Shearing distorts the shape of the image, and as a result, the deformed shape appears as if the rows and columns are sliding up and down relative to each other. Shearing rearranges the positions of pixels according to the following equation: shx =

[0232] The multi-camera control system D100 is applied when the subject D105 has either a higher or lower brightness than the controlled background D104, either in whole or in part. For example, the background of D104 may be illuminated to create this situation. A filtering block D403 passes the brightness information associated with its image data. A single background model D406 represents the expected brightness of this background D104. In practice, the brightness of the controlled background D104 may be variable within the active image region D208, and therefore the background model D406 may store the expected brightness values ​​for all pixels within this active image region D208. A comparison criterion generation process D407 calculates signal noise (more than what can be calculated within the background model) and slight variations in the brightness of the controlled background D104 by modifying each brightness value from the background model D406, and as a result generates the smallest brightness value that can be classified as consistent with the background model D406. For example, if the brightness of the controlled background D104 is higher than the brightness of the target D105, then the processing block D407 reduces the brightness value of each pixel by an amount greater than the expected magnitude and brightness variation of the signal noise.

[0233] In some implementations of system D100, the region D103 is so narrow that it may be modeled as a planar region. The orientation of the plane is parallel to the front and back faces of the cube of dots representing the region D103 in Figure 15. The active image region D208 may be reduced to a single row of pixels in the optional scaling model D404 if two conditions are met: (1) when the target D105 is detected, the background D104 is blocked in all rows and some columns of the active image region D208; and (2) a single set of values ​​in the background model D406 sufficiently characterizes the entire column of pixels in the active image region D208. The first condition is usually met if the active image region D208 is thinner than the target D105. The second condition is met by the implementations of blocks D403, D405, D406, and D407 described above. Applying scaling module D404 reduces the complexity of processes that need to be performed in subsequent processing, as well as the amount of storage required for background model D406.

[0234] The specific implementation of scaling module D404 depends on the specifications of processing blocks D403, D405, D406, and D407. If the brightness of the controlled background D104 is expected to be higher than the brightness of the target D105, as described above, one implementation of scaling module D404 represents each column by the highest brightness within that column. That is, for each column, the highest value within that column is copied to the new array. This process has the advantage that the high-brightness portion of the controlled background D104 does not need to fill the entire controlled background D104.

[0235] The alternative implementation applies to situations where the controlled background D104 is static, i.e., does not involve motion, but its brightness is not limited. A sample source image is included in Figure 19 as an example. In this case, the object may contain, or have values ​​close to, the brightness values ​​found within the controlled background D104, as perceived by the camera. In practice, variations in the brightness of the controlled background D104 (e.g., caused by a user moving forward of the device and thereby blocking some ambient light) may be significant in magnitude relative to the difference between the controlled background D104 and the object D105. Therefore, certain types of filters may be applied in filtering process D403 to produce an unchanging result or a result that does not emphasize the variation in overall brightness, while on the other hand, emphasizing parts of the object D105. A 3x3 previtt filter is commonly used in filtering process D403. Figure 19B shows the result of a 3x3 previtt filter on the image in Figure 19A. In this implementation, two background models D406 may be maintained, one representing high and one low values, and both representing the expected range of values ​​for the filtered pixels, respectively. The comparison criterion generation process D407 then reduces the low values ​​and increases the high values ​​by an amount greater than the expected magnitude and luminance variation of the signal noise. The result is a set of criteria, an example for low values ​​shown in Figure 19C, and an example for high values ​​shown in Figure 19D. These corrected images are then passed through comparison process D408, which classifies pixels that are inconsistent with the controlled background D104 if their values ​​are either lower than the low value criterion (Figure 19C) or higher than the high value criterion (Figure 19D). The result is a binary difference map D409, an example corresponding to Figure 19B shown in Figure 19E.

[0236] Prior implementations have made it possible to use, for example, many existing surface walls, window frames, etc., as the controlled background D104, where these surfaces may have arbitrary brightness, texture, edges, or lines of light fixed to the surface of the controlled background D104. Furthermore, the above implementations have made it possible to use a controlled background D104 that includes, for example, a predetermined pattern or texture, and the above step detects the absence of a pattern in the area where the object D105 obstructs the controlled background D104.

[0237] The difference map D409 stores the locations of all pixels found to be inconsistent with the background D104 by the method described above. In this implementation, the difference map D409 may be represented as a binary image, where each pixel may be in one of two states. These pixels inconsistent with the background D104 are identified or "tagged" by setting the pixels in the corresponding rows and columns of the difference map to one of those states, or by setting the corresponding pixels to the other state.

[0238] An implementation of detection module D410 for detecting the target D105 in difference map D409 is shown in Figure 20. Another scaling module in block D603 provides an additional opportunity to reduce the data to a single-dimensional array of data and may be applied to situations where the orientation of the target D105 does not significantly affect the entire extent of the target D105 within difference map D409. In practice, this applies to many situations where the number of rows is less than or equal to the number of columns normally occupied by the target D105. When applied, the scaling module in block D603 reduces its difference map D409 to a map of one row, i.e., a single-dimensional array of values. In this implementation, scaling module D603 may count the number of tagged pixels in each column of difference map D409. As an example, difference map D409 in Figure 21A is reduced in this manner and shown as graph D709 in Figure 21B. Applying this optional processing step reduces processing requirements and simplifies some of the subsequent calculations.

[0239] Continuing this implementation of detection module D410, it is observed that the pixels tagged in the difference map associated with target D105 (D409 in the example in Figure 31A) roughly form cluster D701, but that the cluster is not necessarily connected. Cluster identification process D604 classifies the pixels (or, if scaling module D603 is applied) based on whether they are members of cluster D701. Various methods exist and may be applied to find clusters in a sample, and the subsequent method is selected based on the simplicity of the process. Note that if target D105 exists, the count of accurately tagged pixels will be greater than the false positive number. Therefore, the central position is expected to be somewhere within target D105. Part of this implementation of cluster identification process D604, when applied to a single-column map (for example, if a scaling module is provided in block D603 or D404), calculates the tag column as part of the central column D702 and cluster D701 (if they are within a predetermined distance D703 corresponding to the maximum number of columns they are to occupy). Part of this implementation of cluster identification process D604, when applied to a multi-row map, adds the tagged pixels to cluster D703 (if they meet the adjacent distance criteria).

[0240] In this implementation, a set of criteria is received by cluster classification process D605 and then assigned to cluster D701 to verify that the cluster is qualified to match what is expected to be the subject D105. Thus, process D605 determines whether cluster D701 should be classified as belonging to the subject D105. Part of this implementation of cluster classification process D605 calculates the count of tagged pixels within cluster D701 and calculates the count of all tagged pixels. The counts within cluster D701 are compared to a threshold to remove false matches in clusters with only a few tagged pixels that are expected to be the subject D105. In addition, the ratio of the pixel counts within cluster D701 relative to the total count is compared to a threshold to further reduce false matches.

[0241] If cluster D701 passes these criteria, the cluster description is fine-tuned in processing block D606 by calculating the center of gravity associated with cluster D701 in processing D607. The central position found by scaling module D603 will be within the range that defines the object D105, but it is not necessarily the center of the object. A weighted average D710, or center of gravity, provides a better measure of the cluster position and is calculated as needed within processing D606, as a subprocess D607. The weighted average D710 is calculated by the following equation: .×..function..×..function.##EQU00003##, where {overscore(x)} is the mean, c is the number of columns, and C[x] is the count of tagged pixels in column x.

[0242] The cluster range D704 may also be calculated within process D606 as needed and shown as process D608. Cluster D703 may contain some false positive outliers, and as part of this implementation, the range may be defined as containing a predetermined percentage of tagged pixels, or, in situations where relatively few pixels are expected to be tagged, containing closely spaced subclusters, i.e., tagged pixels (or columns, if scaling module D603 is applied) that form those tagged pixels (or columns) with untagged neighbors.

[0243] In addition to intermediate and boundary coordinates, the direction of the object D105 may be estimated by calculating the cluster moment, if necessary. This calculation is represented by the cluster direction calculation process in subprocess D609 within process D606.

[0244] In some applications of system D100, object D105 is used as a pointer. In this case, a "pointing end" of object D105 is desired, and if the region D103 contains a sufficient number of rows and that number of rows has not been reduced, it may be determined by a pointing end calculation subprocess in process D606. An example is shown in Figure 21C. Object D105 is normally input to, or restricted from inputting to, the active image region D208 from a known boundary of its region. The pointing end D705 of object D105 (e.g., the user's fingertip) would be part of the cluster D701 furthest from the input region D706 to the active image region D208. Cluster D701 may contain some false positive outliers. Therefore, the pointing edge D705 may be defined as a region D707 within cluster D701 that includes multiple tagged pixels near the furthest boundary of cluster D701, or, in situations where relatively few pixels are expected to be tagged, the furthest tagged pixels that form adjacent subclusters, i.e., those tagged pixels that have tagged neighbors. This subcluster is identified by the subcluster pointing edge process D610, and the location of the subcluster is found in process D611.

[0245] If this implementation continues, the process performed by the smoothing module D612 may be applied as needed to any or all locations found in process D606. Smoothing is the process of combining previously resolved results with each other, so that they move in a stable manner from frame to frame. The weighted average coordinates D710 found by the gravity center determination process D607 depend on many samples and are therefore inherently stable. The range D704 is found by the cluster boundary area determination process D608, and the pointing edge D705 is found by D611, and the coordinates depend on a relatively small number of clusters, and the state of a single pixel may have a significant effect. Since the size of the region indicated by the subject 105 is expected to remain relatively stable, smoothing may be applied to the distance between the ranges D704 measured in relation to the cluster weighted average coordinates D710. Since the shape and orientation of object D105 are expected to change more slowly than the overall position of object D105, smoothing may be applied to the distance of the pointing edge D705 measured in relation to the cluster's weighted average coordinate D710.

[0246] The process used in the central processing of gravity is given by the following equation 1: s(t) = (a × r(t)) + ((1-a) × s(t-1)). In equation 1, the smoothed value at time (s(t)) is equal to the smoothed value at time - 1(t-1) by 1 - scale value (a) × time - 1. This quantity is added to the original value at time t(r(t)) multiplied by a scalar (a) between 0 and 1.

[0247] Referring to Figure 22, the implementation of system D100 utilizes one or more background models D406 (Figure 22) as described above. The implementation of the background model processor or element D405 that generates the background model data D406 is shown in Figure 22. This implementation of background model element D405 automatically and dynamically generates background models, enabling unattended operation of the system.

[0248] The input data D802 is provided by the output of the scaling model 404 for this implementation of the background model element D405. The input is available for all frames and is sampled in sampling process D803. This sample may include the subject D105 and block a portion of the controlled background D104. For each pixel, a range of values ​​may represent the background D104 better than a single value. By including the effect of this range in the background model, the expansion in process D407 may be made tighter. Contributing multiple frames of data to the sample makes this range observable, but also increases the portion of the background D104 blocked by the subject D105 if frames are sampled while the subject D105 is moving. The optimal number of frames to use depends on the expected movement of the subject D105 in the particular application of the system. In practice, for a system tracking a hand, 10 frames representing approximately 0.33 seconds are sufficient to observe the main portion of the range without allowing movement of the subject, thus blocking an unnecessary portion of the background. If a particular background model is compared in comparison process D408 as an upper range of values ​​expected to be consistent with background D104, then the maximum value of each pixel observed in multiple frames may be recorded as a sample value. If a particular background model D406 is compared in process D408 as a lower range of values ​​expected to be consistent with background D104, then the minimum value of each pixel observed in multiple frames may be recorded as a sample value.

[0249] In this implementation of background model element D405, samples from sampling process D803 are added to buffer D804, which has a storage location for n samples, where the oldest sample in the history is replaced. This history therefore contains n sampled values ​​for each pixel. Since the time span d represented in the buffer is rate-dependent, new samples are acquired and added to the history r by equation 2, which is written as equation ##EQU00004## below.

[0250] In this implementation, the central processing block D805 selects a value for each pixel that is specific to the controlled background D104 at the position represented by that pixel, a value determined by the block. One way to select a value specific to the controlled background D104 within processing block D805 is to select the median of n values ​​for each pixel. For any pixel, the n sampled values ​​in buffer D804 may represent the subject D105. The period d is selected such that the subject D105 does not occupy any single pixel of the controlled background D104 for an accumulated period of d / 2 or longer within any time span of d. Therefore, for any pixel, the principal portion of the sample is specific to the background D104, and thus the median of the sampled values ​​is a value specific to the background D104.

[0251] The background model element D405 is adaptive, and any changes to the background D104 are reflected in the output of the central processing block D805, once they are observed within d / 2 time. The system does not need to visualize the entire control background D104 when started; the target D105 may be shown at startup, but it must be observed for d time before the sample provides output. Restrictions may be applied as needed, such that the target D105 is absent when the system starts, in which case the first observed sample value may be copied to all n samples in buffer D804, allowing the system to generate output more quickly.

[0252] The duration for which any one pixel of the controlled background D104 is blocked by the target D105, and therefore the duration d, depends on the specific use of the system. The number of samples n can be estimated with respect to the memory buffer and available processing power.

[0253] The preceding discussion describes one implementation of obtaining the position of object D105 within and relative to images acquired by cameras D101 and D102. Once object D105 is successfully detected and its coordinates are found in both camera fields D205 and D206 by detection modules D308 and D309 in Figure 17, the combination of these coordinates is then sufficient to recover the position of object D105 within the range of object D103. In the implementation described in Figure 17, the position of object D105 is calculated in combination module D312.

[0254] Referring to Figures 23A and 23B, the implementation of the combination module D312 is shown. For each camera D101 and D102, the position D902 of the object D105 on the camera image plane D904 is converted to an angle D905, which is referred to herein as β(.β), measured on a reference plane, and its normal is defined by the axis of rotation of cameras D101 and D102. (In practice, the axes are not exactly parallel and are not precisely defined on a single plane, however the processing described herein allows for such errors). By approximating cameras D101 and D102 as ideal pinhole models of cameras, their angle (.β) is approximated in relation to the vector D906 that defines the orientation of the cameras.

[0255] As shown in Figure 23A, Equation 3 shows the following approximate calculation: β × ##EQU00005##. To approximate the angle β(β), the inverse tangent is applied to the amount of focal length (f) divided by the position p on the image plane projected onto the intersection of the reference plane and the image plane.

[0256] For maximum accuracy, essential camera parameters (principal point and image scale position) and radial distortion caused by the lens should be corrected by transforming the distorted position (as represented by relative position information D310, D311) to an ideal position. More specifically, the ideal position is the position on the image plane D904 onto which the object D105 is projected, assuming that cameras D101, D102 have the characteristics of an ideal pinhole camera (where equation 3 generates the exact angle). A set of correction equations is presented in Z. Zhang, A Flexible New Technique for Camera Calibration, Microsoft Research, http: / / research.microsoft.com / .about.zhang, and is incorporated by reference. For many applications of this system, it is understood that approximations provide sufficient accuracy without the above-described corrections.

[0257] Continuing the description of combination module D312, as shown in Figure 23B, the reference vector D907 is defined such that it passes through the positions of both cameras D101 and D102 on the reference plane, where the reference plane is defined such that the axis of rotation of the camera defines the normal to the reference plane. The angle D908 at which the camera is rotated is measured relative to the reference vector D907.

[0258] The formula for measuring angles is given by Equation 4: α = β0 + β. The measurement of angle α(α) is equal to angle β_not(β0) and angle β(β).

[0259] Equation 4 is applied to measure the angle D909 of the object D105 with respect to the reference vector D907. This angle is referred to herein by the symbol α(.α.). The angle αD909 for each camera D101 and D102, and the length of the reference vector D907 are sufficient to find the position of the object D105 on the reference plane by equations 5 and 6.

[0260] Equation 5 is given by: Equation: .×. .×..×. .×..α..×..×. .×..α. .×..×. .×..α..×. .×..α. ##EQU00006## The offset (y) is equal to the reciprocal of the tangent of the angle (.αA) relative to camera A101, the tangent of the angle (.αB) relative to camera B D102 multiplied by the vector length D907(w), the tangent of the angle (.αA) relative to camera A D101, and the tangent of the angle (.αB) relative to camera B D102.

[0261] Equation 6 calculates the offset of the object (xA) as follows: .×. .×..α.##EQU00007##. In Equation 6, the offset (xA) is measured by the offset from Equation 5(y), which is divided by the tangent of the angle (.αA) relative to camera A D101.

[0262] The position of object D105 on an axis perpendicular to the reference plane may also be found by equation 7(.×. .×.##EQU00008##), which is applied to the position in each image using the distance of object D105 from the camera.

[0263] In Equation 7, position (z) is calculated as the position (p) on the image plane projected onto a vector in the image plane that is orthogonal to its use in Equation 3, which is obtained by dividing the position (z) by the focal length (f) multiplied by the distance of the object D105 from the camera (l).

[0264] These relationships provide the coordinates of the object D105 relative to camera A D101. Once the position and size of the region D103 relative to camera A D101 are known, its coordinates may be transformed so that they relate to the regions D103 and D312 in Figure 17.

[0265] Smoothing may be applied as needed to these coordinates in the fine-tuning module D313 of the implementation of this system shown in Figure 17. Smoothing is the process of combining previously solved results and their results so that the motion is stable from frame to frame. One method of smoothing for these particular coordinate values ​​(xA, y, z found by the combination module D312) is described in the present invention. The components of the coordinate values ​​related to the object D105, namely x, y and z, are smoothed independently and dynamically. The degree of dampening S is calculated by equation 8, where S is dynamically and automatically adjusted in response to the change in position and is calculated as follows: .×. .×..ltoreq(≦).α..×. .×..α..×..×. .×..×. .×..α..×. .×.<<.×. .×.gtoreq(≧)..×..×..function..function.##EQU00009##. In equation 8, s(t) is the smoothed value at time t, r(t) is the raw value at time t, DA and DB are thresholds, and SA and SB define the degree of dampening.

[0266] As shown in Figure 24, the two distance thresholds DA and DB define three ranges of motion. Positional changes and motions below DA are significantly dampened by SA (D1001), thereby reducing the tendency of values ​​to switch back and forth between two adjacent values ​​(a side effect of separate sampling of the image). Positional changes greater than DB are slightly dampened, or not dampened, by SB (D1002). This reduces or eliminates the lag and vagueness introduced in some other smoothing procedures. The degree of dampening varies between DA and DB, i.e., for motion in the region shown as D1003, resulting in less pronounced transitions between slight and significant dampening. The scalar a applied to Equation 1 is found by Equation 9 as follows: In Equation 9, the scalar (a) is limited to being equal to or greater than 0 and less than or equal to 0, the damping value of S is found by Equation 8, and e is the elapsed time from the previous frame.

[0267] The coordinates D314 of the object D105, if found, are typically carried to another process, such as the user application program D316. They may, if executed, be carried to another process running on the same image processor D106, or to a separate computing device, as described above. The method by which the data is carried to the application program D316 may include emulation of conventional user input devices (including mouse and keyboard), allowing the system to provide existing control functions within the application program D316. The coordinates D314 of the object D105 may be calculated for every video frame captured by the camera, where a single video frame is typically captured 30 or more times per second. As a result, there is virtually no delay between user actions and the application's response.

[0268] During normal operation of the system, the application program D316 provides feedback to the user by displaying a visual representation of an indicator on the video display D107. The indicator is moved so that its position and movement mimic the movement of the object D105.

[0269] In one variation of this form of user interface, an indicator, such as a mouse pointer, is displayed in front of other graphics, and its movement is mapped to a two-dimensional space defined by the screen surface. This form of control is similar to that provided by a computer mouse, such as those used with the Microsoft.RTM.Windows.RTM. operating system. An exemplary feedback image of an application using this type of control is shown in D1102 in Figure 25A.

[0270] Referring to Figure 25A (and briefly to Figure 17), the image processor D106 also includes an optional coordinate remapping process D317 (Figure 17). The coordinate remapping process D317 can operate to remap the global positioning existence and location coordinates D314 (related to the object D105) to a position overlaid on the image D1102, using the equivalent equations for the x-coordinate and the y-coordinate, as shown below: <.ltoreq(≦)..ltoreq(≦).> ##EQU00011##.

[0271] In equation 10, xh is the coordinate position D314 related to object D105, xc is the cursor position on the screen mapped to 0-1, and bl and br are the positions of the left and right ranges of the sub-region within the region D103. As shown in Figure 25B, the entire region of display D1102 is represented by the sub-region D1103 which is contained within the entire region D103. A position within the sub-region D1103 (e.g., A D1105) is linearly mapped to a position within display D1102 (e.g., D1106). A position outside the sub-region D1103 but still within the region D103 (e.g., position B D1107) is mapped to the nearest position on the boundary of display region D1102 (e.g., D1108). This reduces the likelihood that the user might unintentionally remove the object D105 from the lower area (usually with their hand or finger) while attempting to move indicator D1101 near the edge of the display.

[0272] In a situation where the area D103 is directly in front of the video display D107, the lower area D1103 may be defined to adjust to the video display D107, so that the indicator D1101 appears to be aligned with the object D105. If the area D103 is relatively thin, for example less than 5 cm, and the lower area D1103 is defined in this way, then the system then approximates user interaction, i.e., "touchscreen," without being limited by the size of the video display D107 and without requiring direct contact between the user and the surface of the video display D107 (for example, the video display and the user may be on opposite sides of the window). As understood, the system D100 can be used with a variety of video display sizes and may include not only computer monitors (CRT or LCD displays) but also rear-projection television monitors, large flat-screen LCD monitors, and forward-projection presentation systems.

[0273] In situations where the region D103 is not directly in front of the large video display D107, the active image region D208 is sufficiently deep, and the direction of the object is determined in the direction calculation process D609, the vector may extend from the position of the object to the video display D107 using the angle of direction for detecting the position on the video display that the user is "pointing" to.

[0274] However, the active image region D208 is often not deep enough in processing block D609 to accurately calculate its orientation. In these situations, if region D103 is not directly in front of the large video display D107 and its orientation is not calculated, equation 10 may apply, where the lower region D1103 is smaller than the video display. The processor then maps the absolute position of the object D105 to a position indicator, and as a result, the movement of the object D105 is estimated as a large movement of the position indicator on the video display, thereby making the entire area of ​​the video display easily reachable by the user (for example, the lower region D1103 may be defined to be at most 750 mm wide and a corresponding height, and to be easily reachable by most users). When set in this way, the system still continues to provide the user with the feeling of "pointing at the screen".

[0275] In another variation of this form of user interface, the user moves the display of an indicator within a three-dimensional virtual environment (examples are shown in Figures 26A and 26B). The virtual environment may be generated using projection transforms, so that the depth of the virtual environment is indicated by the image shown on the video display D107. Techniques for generating this type of virtual environment include OpenGL. Equation 10 is used to remap the x, y, and z coordinates (the sub-region 1103 becomes, for example, a cube).

[0276] Applications controlled by movable objects on screen indicators (e.g., Figures 25A, 26A, and 26B), whose control is described above, generally display a graphic representation of data or interactive elements (e.g., a button D1109 or an object display D1202). The user is expected to position indicator D1101 over one of these objects, or, if a three-dimensional virtual environment is presented, to touch or interact with the object. For two-dimensional interfaces, this state may be detected by comparing the remapped indicator position D1106 with the boundary of the object's graphic representation (e.g., D1110), and this state is true if the indicator's position is within the object boundary. For three-dimensional interfaces, this state may be detected by comparing the boundary D1203 of the entire indicator D1101, or, if finer control is required, a portion of the indicator, with the boundary D1204 of object D1202. The user optionally receives feedback indicating that the cursor is positioned over the object. The feedback may take various forms, including audio cues and / or graphic changes to either or both the cursor and the object. The user may then activate, manipulate, or move the object under the cursor. The user is expected to indicate their intention to activate, manipulate, or move the object by performing a gesture.

[0277] Optionally, the motion of the target object D105 may be interpreted and classified by the gesture detection module D315, as described above with respect to Figure 17. The gesture detection process D315 may utilize data formed from any component of the system. Final coordinates D314, image coordinates D310 and D311, or combinations thereof D310, D311 and D314 may be sampled over time and provided as input to the gesture detection process D315. By using this data as input to the gesture detection process D315, various gestures (e.g., "hovering" and "poking") were successfully detected.

[0278] In a scenario where the application state (i.e., whether indicator D1101 is over button D1109) is known and communicated to the gesture detection module D315, one gesture the user performs to indicate the intention to activate an object (e.g., screen objects D1109, D1202) under cursor D1101 is to hover the cursor over the object (e.g., D1109, D1202) for a longer duration than predetermined. If the application state does not change over the predetermined duration, this gesture performed by the user is detected by monitoring the application state and triggering the gesture. There is no need to develop an application specifically for the multi-camera control system D100, because there are techniques that can discreetly monitor the application state (by setting a "hook" using the Windows SDK function "SetWindowsHookEx" in the Windows operating system) and emulate a mouse "click" (using the Windows SDK function "SendInput" in the Windows operating system).

[0279] In some scenarios, the state of the application may not be available and may not need to be monitored. In this case, some exemplary gestures indicating the intention to activate an object under cursor D1101 (e.g., screen objects D1109, D1202) may be keeping the hand still ("hovering") or rapidly poking the hand back and forth.

[0280] The method for detecting "hovering" is to maintain a history of the position of the target object D105, which includes all records of its position and state over a predetermined duration, ending with the most recent sample. This duration represents the minimum duration for which the user must keep their hand still. The minimum and maximum positions, which are separate for each of the three dimensions (x, y, z), are found in the history. A "hovering" gesture is reported if the target object D105 is within the target region D103 in all samples of the history, and the distance between the minimum and maximum is within a predetermined threshold for each of the three dimensions. These distance thresholds represent the maximum amount the target object D105 can move, as well as the maximum amount of change (or "jitter") that is to be introduced into the hand position by various components of the system. When the system emulates a mouse as described above, a common way this gesture is reported is by emulating a mouse "click". Gestures representing further mouse operations, "double-click" and "drag," have also been detected and emulated.

[0281] Optionally, gestures independent of the indicator's position relative to the object may be detected and given meaning by the application, depending on or independent of the application's state. Applications using this style of interaction generally do not explicitly use or display the position D317 or other positions of the target object. These applications can be controlled entirely or primarily by the system's interpretation of positions. These applications also do not need to be developed specifically for this system, because the interpretation performed by this system can be used to simulate actions that may be performed on conventional user input devices (e.g., keyboards or joysticks).

[0282] Many useful interpretations depend directly on the absolute position of the object D105 within the area D103 (or, in an equivalent way, on the indicator position D1105 within the sub-area D1103). One way to perform these interpretations is to define a box, plane, or other shape. The state is triggered ON if it is found that the position of the object D105 (e.g., the position defined by block D314, or the position defined by the coordinates remapped from remapping process D317) is within the first box (or beyond the boundary defined by the first plane) and was not found in the previous observation (because it was elsewhere in the area D103 or was not detected). This state is maintained until the hand position is no longer found within the second box (or beyond the boundary defined by the second plane), at which point the state is triggered OFF. The second box must contain the entire first box and is generally larger. The occurrence of a state where the gesture is accidentally triggered on and off when it is detected that the target object D105 is near the box boundary, and where very small motion or slight noise in the image signal causes position D317 to drift in and out of the box, is mitigated by using a larger box. Depending on the use of the gesture, one of three ways of interpreting this state is commonly used. In one method, the gesture directly reflects the state through on and off triggers. A keyboard key or joystick firing button, if emulated, is "pressed" when the state is triggered on and "released" when the state is triggered off. In another method, the gesture is triggered only by a transition from off to on. If a keyboard key or joystick button is emulated, the key is "clicked". The duration and off state are not reported to the application but are maintained so that the gesture is not repeated until the state is triggered off. As a result, each instance of the gesture requires an intent clearly defined by the user.A third method involves triggering the gesture by transitioning from an on-to-off state, and periodically re-triggering the gesture at predetermined intervals as long as the state remains on. This emulation is similar to how holding down a key on a keyboard repeats characters in some applications.

[0283] One way in which a box or plane can be defined within the target region D103 for the above technique is as follows: By defining a first plane (D1501 in Figure 27A) and a second plane D1502 that divide the target region into a "firing" region D1503 and a "neutral" region D1504 (as mentioned above, the gesture reported when the target object D105 is within the region D1505 between the planes depends on the object's previous position), the above technique can detect a target object D105 (generally a hand) being "pushed" forward, and this gesture is one gesture (e.g., firing a weapon in a video game) to emulate a firing button on a joystick or to cause an application to react in relation to the pressing of a joystick button.

[0284] Another technique for defining boxes or planes within the target region D103 for the above technique is as follows: As shown in Figure 27B, the left, right, top, and bottom portions of the target region D103, which partially overlap at the corner region, are separated to define the first type of planes D1506, D1507, D1508, and D1509. The second type of planes are labeled D1510, D1511, D1512, and D1513. Each pair of the first and second planes is processed independently. This combination of planes emulates four directional cursor keys. Here, the hand at the corner triggers two keys which are generally interpreted by many applications as four secondary 45-degree (diagonal) directions. By emulating the keyboard cursor in this way, various existing applications can be controlled by system D100. Applications (including Microsoft® PowerPoint®) respond to emulated cursor keys (e.g., up and down arrow keys) for example, by advancing to the next or previous slide in a presentation sequence.

[0285] Another way to emulate inconspicuous directional control applies to applications where four 45-degree directional states are expected to be explicitly represented. As illustrated in Figure 27C, boxes D1514, D1515, D1516, and D1517 are defined for each of the four primary (horizontal and vertical) directions, and further boxes D1518, D1519, D1520, and D1521 are defined for each of the secondary 45-degree (diagonal) directions. For clarity, only the first type of boxes are illustrated. Gaps are placed between these boxes. Figure 27D illustrates how adjacent boxes are defined. The gap between the first type of boxes D1522 and D1523 ensures that the user intentionally places the target object D105 into the box, while gap D1524 is filled by partially overlapping the second type of boxes D1525 and D1526. As a result, the system reports the previous gesture until the user explicitly intends to move the target object D105 to an adjacent box or the central neutral area. This combination of buttons can be used to emulate an 8-way joystick pad.

[0286] A broader range of gesture types rely on motion, either instead of or in addition to position. One example is the "swipe left" gesture, which communicates to an application that you want to return to the previous page or state. Across keyboard and mouse emulations, this gesture may be used to control information presentation software (particularly Microsoft® PowerPoint®) to go to the previous page of a presentation sequence. Across keyboard and mouse emulations, this gesture causes a web browser to perform an action associated with a "back" button. Similarly, the "swipe right" gesture is one way to communicate to an application that you want to advance to the next page or state. For example, this gesture causes presentation software to advance to the next slide of a presentation sequence, and browser software to advance to the next page.

[0287] One way to detect a "swipe left" is as follows: A thin stripe along the leftmost part of the target area D103 is defined as the leftmost area. The position of the target object D105 (for example, the position defined by block D314, or the position defined by the remapped coordinates from the remapping process D317) is represented as one of three states: 1. The target object exists and is not inside the leftmost area. 2. The target object exists and is inside the leftmost area. 3. The target object is not located within the hand detection area.

[0288] The transition from state 1 to state 2 described above causes the gesture detection module D315 to enter a state where it starts a timer and waits for the next transition. If a transition to state 3 is observed within a predetermined duration, it is reported that a "swipe hand to the left" gesture was performed. This technique is generally reproducible for the right, upper, and lower edges, and since the hand position is found in three dimensions, it is also reproducible for "pulling the hand back".

[0289] Various gesture detection techniques have been described. Furthermore, other gesture detection techniques (e.g., hidden Markov layers) are described in the research literature and may be applied to various implementations of system D100 described herein.

[0290] Referring again to Figures 15 and 17, another embodiment of the multi-camera control system D100 will be described in further detail. While Figure 15 shows a two-camera system, it should be understood that the image processing processor D106 can be configured to receive input from more than two cameras and may include four or more video cameras for specific applications. In the four-camera embodiment, the components D304-D311 in Figure 17 are reproduced to support two additional cameras. In addition, the combination module D312 is configured to receive presence and location information (similar to data D310 and D311) associated with four sets of cameras related to the object D105 being tracked. The techniques and formulas described above (in particular formulas 5 and 6) can be applied to the additional pairs of cameras, where the output of the combination module D312 is the average of all positions from each of the pairs of cameras. The gesture detection module D315 is similarly reconfigured to receive presence and location information D310, D311 associated with four sets of cameras from two additional detection modules (similar to D308 and D309), which are substantially the same as detection modules D310 and D311.

[0291] The output from the image processing processor 106 (in this case, including processed object position coordinates and gesture information related to the four cameras) can be used by another process or user application program 316. The formulas and geometry (described above) used to calculate the coordinate information related to the target object 105 from two additional cameras are also used.

[0292] In one embodiment using four cameras, two additional cameras are positioned at the two bottom corners of a controlled background D104 and oriented so that the target area D103 is within the field of view D205 of each camera. The advantage of the four-camera system is that the position of the target object D105 can be tracked with greater precision. Thus, the application program may include more screen objects at a higher density on the video display D107 because the improved tracking accuracy allows very nearby objects to be correctly selected by small movements of the target object D105. Furthermore, the two additional cameras reduce errors in tracking the target object D105 when a portion of the target object D105 is obstructed in the field of view D205 related to one or more other cameras.

[0293] (Neutral position of the device) According to one general embodiment, a method is disclosed. The method includes the steps of determining a neutral position of the device with respect to at least a first axis, and measuring the angular displacement of the device with respect to at least a first axis, wherein the device includes a first control associated with at least a first plurality of output signals. The method also includes the steps of receiving a selection of the first control and outputting one of the first plurality of output signals based on at least the selection and the angular displacement.

[0294] The embodiment may include one or more of the following features. For example, the neutral position of the device may be determined with respect to at least a second axis (orthogonal to the first axis), where the angular displacement may include the first axis component and the second axis component. Furthermore, the neutral position of the device may be determined with respect to at least a third axis (orthogonal to the first and second axes), where the angular displacement may include the third axis component. The first axis, second axis and / or third axis may intersect within the device.

[0295] The first control may relate to at least three output signals, or at least nine output signals, where each of the multiple output signals may correspond to a character such as an alphanumeric character. The method may further include the steps of displaying the output signals and / or displaying an indication of angular displacement. The method may also further include the steps of defining a plurality of tilt regions with respect to the first axis, where one of the first plurality of output signals is also an output based on the plurality of tilt regions. The angular displacement of the device with respect to the first axis may be measured as 0 degrees, and the first tilt region includes the angular displacement of 0 degrees. Alternatively, the first tilt region may be defined as a region including about -30 degrees to 0 degrees with respect to the first axis, and the second tilt region may be defined as a region including about 0 degrees to +30 degrees with respect to the first axis. In a further embodiment, if the angular displacement is within the first tilt region when the selection is received, a first output signal may be output. If the angular displacement is within the second tilt region when the selection is received, a second output signal may be output. If the angular displacements are within the third or fourth inclination region when the selection is received, the third or fourth output signal may be output.

[0296] The above method may also define a plurality of first axis tilt regions with respect to the first axis, and a plurality of second axis tilt regions with respect to the second axis, where one of the first plurality of output signals may also be output based on the plurality of first axis tilt regions and / or the plurality of second axis tilt regions. If, when the selection is received, the first axis component is in the first first axis tilt region and the second axis component is in the first second axis tilt region, the first output signal may be output. If the first axis component is in the second first axis tilt region and the second axis component is in the first second axis tilt region, the second output signal may be output. If the first axis component is in the second first axis tilt region and the second axis component is in the second second axis tilt region, the third output signal may be output. and / or, if the first axis component is in the second first axis tilt region and the second axis component is in the second second axis tilt region, the fourth output signal may be output.

[0297] Alternatively, in another embodiment, when a selection is received, if the first component is in the first first axis tilt region and the second axis component is in the first second axis tilt region, a first output signal may be output. If the first component is in the first first axis tilt region and the second axis component is in the second second axis tilt region, a second output signal may be output. If the first component is in the first first axis tilt region and the second axis component is in the third second axis tilt region, a third output signal may be output. If the first component is in the second first axis tilt region and the second axis component is in the first second axis tilt region, a fourth output signal may be output. If the first component is in the second first axis tilt region and the second axis component is in the second second axis tilt region, a fifth output signal may be output. If the first component is in the second first axis tilt region and the second axis component is in the third second axis tilt region, a sixth output signal may be output. If the first component is in the third first axis tilt region and the second axis component is in the first second axis tilt region, a seventh output signal may be output. If the first component is in the third first axis tilt region and the second axis component is in the second second axis tilt region, an eighth output signal may be output. And / or, if the first component is in the third first axis tilt region and the second axis component is in the third second axis tilt region, a ninth output signal may be output.

[0298] In another general embodiment, an apparatus is disclosed. The apparatus includes a tilt sensor configured to determine the neutral position of the apparatus with respect to at least a first axis and further configured to measure the angular displacement of the apparatus with respect to at least the first axis. The apparatus also includes at least a first control associated with a first plurality of output signals and a processor, the processor configured to receive a selection of the first control and further configured to output one of the first plurality of output signals based at least on the selection and the angular displacement.

[0299] The embodiment may include one or more of the following features. For example, the first and second axes may intersect at the center of the device or at the periphery of the device. The device may further include at least second to tenth controls, each associated with a plurality of output signals from second to tenth. The first control may be a button, and / or the device may be a telephone. The displacement signal may be measured using a tilt sensor, and the tilt sensor may be a gyroscope. The device may further include a display configured to display the output signals, and / or a display configured to display an indication of angular displacement, and the device may further include a keyboard configured for inputting selections.

[0300] In another general embodiment, a computer program product explicitly stored on a computer-readable medium is disclosed. The computer program product is operable to cause a computer to perform operations including determining the neutral position of the device with respect to at least a first axis and measuring the angular displacement of the device with respect to at least a first axis, wherein the device includes a first control associated with at least a first plurality of output signals. The computer program product is also operable to cause a computer to perform operations including receiving a selection of the first control and outputting at least one of the first plurality of output signals based on the selection and the angular displacement.

[0301] In another general embodiment, a telephone device is disclosed. The telephone device includes a tilt sensor configured to determine the neutral position of the telephone device with respect to at least a rolling axis and further configured to measure the angular displacement of the telephone device with respect to at least the rolling axis. The telephone device also includes at least 1 to 8 buttons, each associated with at least 4 alphanumeric characters. Furthermore, the telephone device includes a processor configured to receive a selection of the 1st button and further configured to output at least one of the 4 alphanumeric characters based at least the selection and angular displacement.

[0302] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features will become apparent from the description and drawings and the claims.

[0303] Figure 28 shows the appearance of the device according to one exemplary embodiment with the device in a neutral position. The hardware environment of the device E100 includes a keypad with at least a first control E102 for inputting text data and user commands into the device E100, a display E105 for displaying text and images to the user, and an indicator (e.g., a tilt indicator E106) for displaying indications of angular displacement or tilt orientation with respect to at least one axis.

[0304] Display E105 displays graphics, images, and text, and includes a user interface for software applications used in this embodiment, as well as an operating system program necessary for operating the device E100. The user of device E100 uses a first control E102 to input commands and data for operating and controlling the operating system program and application programs.

[0305] Display E105 is configured to display a GUI to the user of device E100. A speaker may be present, which may generate voice and voice data received from application programs running on device E100 (e.g., voice from another user generated by a telephone application program) or ringtones generated by a ringtone application program. A microphone may be used to capture voice data generated by the user, for example, when the user is making a call with another user through device E100. Furthermore, a tilt indicator E106 is configured to indicate the angular displacement or tilt orientation of device E100, providing visual feedback to the user of device E100 and informing the user of the tilt orientation used to interpret control selections.

[0306] The operation of device E100 is based on the orientation of the device in two states: a "neutral" position and a "selected" position corresponding to the position of the device before, simultaneously with, or after the selection of the first control E102. More specifically, as will be fully explained below, the output signal from device E100 depends on the angular displacement between the neutral position and the selected position with respect to at least one axis, where the angular displacement has an angular displacement component for each axis in question.

[0307] Figure 28 shows, for example, the device E100 in one possible three-axis neutral position. In particular, the orthogonal X, Y, and Z axes intersect at the center of the device E100. Here, the X axis extends parallel to the longitudinal direction of the device E100. According to this exemplary neutral position, rotation around the X axis achieves rolling motion, rotation around the Y axis achieves pitching motion, and rotation around the Z axis achieves yawing motion. These rolling, pitching, and yawing motions are generally referred to as “tilting” motions in this specification.

[0308] The determination of the number of axes to be considered, and the position and orientation of the axes relative to the device E100, are device-specific and application-specific determinations, and no limitation of these features is inferred in the following description. For example, if it is undesirable or impossible to operate the device in yawing motion, or if the number of output signals can be effectively controlled using motion around one or two axes, the neutral position of the device may be determined with respect to only these one or two axes. Furthermore, at least one axis does not have to intersect the device E100, or at least one axis may extend along the periphery or edge of the device E100. In addition, one of the axes may extend parallel to the longitudinal direction of the device E100, or at an angle to the longitudinal direction of the device E100. In any case, the neutral position is aligned using an axis relative to the Earth (e.g., the magnetic north axis or the true north axis, or an axis pointing to the center of the Earth or the horizon), or an axis relative to the user, the device, or other axes.

[0309] For telephone communications, a single-axis neutral position is provided when angular displacement is measured with respect to rolling rotation around the X-axis, and a two-axis neutral position is provided when angular displacement is measured with respect to rolling and pitching rotation around the X-axis and Y-axis, respectively. In either case, the X-axis and Y-axis intersect at the center of the device, and the X-axis extends in the longitudinal direction parallel to the longitudinal direction of the device. Other neutral position orientations are also possible.

[0310] When typing on a device like a telephone, users typically hold the device at a positive (upward) pitch angle while looking at the display. In this regard, the X-axis of the telephone in the neutral position may be defined as a similar upward angle, and consequently, flattening the telephone's angle with respect to the ground may be registered as a pitched forward tilt motion. In other cases, naturally, the X-axis parallel to the ground is the "neutral" X-axis position.

[0311] In Figure 28, the device E100 is depicted as a mobile phone, but in further embodiments, the device E100 may include a desktop PC, laptop, workstation, midrange computer, mainframe computer, handheld computer, tablet computer, personal digital assistant ("PDA"), or another type of embedded system (e.g., a computer keyboard or remote control).

[0312] Figure 29 shows an example of the internal architecture of the embodiment of Figure 28. The computing environment includes a processor E200 on which computer instructions, including an operating system or applications, are processed; a display interface E202 that provides a communication interface and processing functions for generating graphics, images, and text on a display E105; a keypad interface E204 that provides a communication interface to a keypad including a first control E102; a tilt sensor E206 for measuring the angular displacement of the device E100 with respect to at least a first axis; an indicator interface E208 that provides a communication interface to an indicator including a tilt indicator E106; random access memory ("RAM") E210 in which computer instructions and data for processing by the processor E200 are stored in a volatile memory device; and basic system functions. The system includes read-only memory ("ROM") E211 in which immutable low-level system code or data (e.g., basic input / output ("I / O"), startup, or receiving keystrokes from a keypad) is stored in a non-volatile memory device, and optionally memory E220 or other suitable types of memory (e.g., random access memory ("RAM"), read-only memory ("ROM"), programmable read-only memory ("PROM"), erasable PROM ("EPROM"), electrically erasable PROM ("EEPROM"), magnetic disk, optical disk, floppy disk, hard disk, removable cartridge, flash drive) in which files including the operating system E230, application programs E240, and data files E246 are stored. The components and processor E200 communicate with each other via bus E250.

[0313] RAM E210 interfaces with bus E250 to provide rapid RAM storage to processor 200 during the execution of software programs such as operating systems, application programs, and device drivers. More specifically, to execute a software program, processor E200 loads the computer-executable process from the memory medium into the fields of RAM E210. The data is stored in RAM E210 and accessed by processor E200 during execution.

[0314] As further shown in Figure 29, the storage device E220 stores computer-executable code for the operating system E230, application programs E240 (e.g., word processors, spreadsheet software, presentations, graphics, image interpretation training, games, or other applications), and data files E246. While the above embodiments can be used, the functionality provided by this disclosure can also be implemented as a dynamic link library ("DLL") or as a plug-in to other application programs (e.g., an internet web browser (e.g., the Microsoft® Internet Explorer web browser)).

[0315] Processor E200 is one of many high-performance computer processors, and without departing from the scope of this disclosure, includes, but is not limited to, INTEL® or AMD® processors, POWER PC® processors, MIPS® Reduced Instruction Set Computer ("RISC") processors, SPARC® processors, HP ALPHASERVER® processors, ACORN® RISC Machine ("ARM®") architecture processors, or any other proprietary computer processor for computers or embedded systems. In additional devices, Processor E200 of Device E100 is a plurality of processing units, including a multi-CPU configuration found in high-performance workstations and servers, or a multi-scalable processing unit found in mainframes.

[0316] Operating System E230 supports Microsoft® Windows NT® / Windows® 2000 / Windows® XP workstations, Windows NT® / Windows® 2000 / Windows® XP servers, various UNIX®-based operating systems (including IBM® AIX® for workstations and servers, Sun® SUNOS® for workstations and servers, Intel® LINUX® for CPU-based workstations and servers, HP® HP UX Workload Manager® for workstations and servers, SGI® IRIX® for workstations and servers, Digital Equipment Corporation VAX / VMS for computers, HP® ALPHASERVER-based OPENVMS® for computers, and MAC OS® X for POWERPC®-based workstations and servers), SYMBIAN OS® for mobile devices, and Windows It may be MOBILE® or WINDOWS CE®, PALM®, NOKIA®OS ("NOS"), OSE® or EPOC®, or any operating system for which you can claim ownership for a computer or embedded system. The application development platform or framework for operating system E230 may be BINARY RUNTIME ENVIRONMENT FOR WIRELESS® ("BREW®"), Java Platform Micro Edition ("Java ME") or Java 2 Platform Micro Edition ("J2ME®"), PYTHON®, FLASH LITE®, or MICROSOFT®NET Compact.

[0317] The tilt sensor E206, as described later, detects the orientation of the device E100 and is a gyroscope, optical sensor, and / or other type of tilt sensor. The optical sensor may be used, for example, to detect the orientation of the device E100 using the optical flow of a series of images from a camera incorporated in the device E100, and to determine the motion and orientation of the device E100. The optical flow indicates the apparent relative velocity of features within the range of the series of images. Since the optical flow is related to the camera, the apparent velocity of features within the camera's field of view can be obtained from the camera's motion. The camera's motion is calculated from the apparent velocity of features within the camera's field of view. Position or orientation is also calculated in relation to a neutral position over a long period of time. Although the tilt sensor E206 has been described as an optical sensor using an optical flow method to track the tilt or gradient of the device E100 using a camera, in other embodiments, the tilt or gradient of the device E100 may be tracked without using an optical flow method, for example, using an accelerometer.

[0318] The computer-readable memory medium stores information within the device E100 and is volatile or non-volatile. The memory may be capable of providing large-capacity storage to the device E100. In various different embodiments, the memory may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive. Figures 28 and 29 illustrate one possible embodiment of a computing system that executes program code or program steps or process steps, but other types of computers or devices may be used.

[0319] Figure 30 is a flowchart illustrating a method according to another exemplary embodiment. Briefly, the method includes the steps of determining a neutral position of the device with respect to at least a first axis, and measuring the angular displacement of the device with respect to at least the first axis, wherein the device includes a first control associated with at least a first plurality of output signals. The method also includes the steps of receiving a selection of the first control and outputting at least one of the first plurality of output signals based on the selection and the angular displacement.

[0320] More specifically, method E300 is initiated (step ES301), and multiple inclination regions are defined with respect to the first axis (step ES302). As will be described in more detail below, the output of the output signal is based at least on the angular displacement of the device when the first control is selected. According to one embodiment, when the control is selected, an output related to the inclination region is output if the angular displacement is within a specific inclination region or band of angles.

[0321] Figures 31A to 31D illustrate several exemplary slope regions with respect to a hypothetical neutral axis labeled "n-axis," where "neutral" represents the neutral X, Y, and / or Z axes. Each of the X, Y, or Z axes may have individually determined slope regions. A common slope region definition can be applied to multiple axes. Alternatively, an axis may not have a defined slope region.

[0322] Figure 31A illustrates examples of two inclination regions defined with respect to the neutral axis. Angular displacements of approximately -90 degrees to 0 degrees with respect to the neutral axis fall within region E401, while angular displacements of approximately 0 degrees to approximately 90 degrees with respect to the neutral example fall within region E402. Angular displacements of approximately 91 degrees to -91 degrees (indicating that the device is inverted) do not fall into either region, and an angular displacement of exactly 0 degrees falls within either region E401 or region E402.

[0323] If the neutral axis represents the X-axis, angular displacement in region E401 may be due to negative (leftward) rolling of the device, and angular displacement in region E402 may be due to positive (rightward) rolling of the device. If the neutral axis represents the Y-axis, angular displacement in region E401 may be due to negative (forward) pitching of the device, and angular displacement in region E402 may be due to positive (rearward) pitching of the device. If the neutral axis represents the Z-axis, angular displacement in region E401 may be due to negative (counterclockwise) yawing, and angular displacement in region E402 may be due to positive (clockwise) yawing. Two tilt regions are shown, but any number of tilt regions may be defined, depending largely on the sensitivity of the tilt sensor, the number of output signals associated with each control, and the user's ability to distinguish small angles when operating the device.

[0324] In any case, the signal output by the device depends on the angular displacement and the tilt region. For example, the device outputs the first of several signals when the device's angular displacement is in the first region, and outputs the second of several signals when the device's angular displacement is in the second region, even if the same control is selected in both situations. Figure 28 illustrates that regions E401 and E402 include a bandwidth of ±90 degrees, but in a similar embodiment, tilt region E401 defines a region including approximately -30 degrees to 0 degrees with respect to the neutral axis, and tilt region E402 defines a region including approximately 0 degrees to +30 degrees with respect to the neutral axis.

[0325] Figure 31B shows four defined tilt regions around the neutral axis, with dead space between 0-degree regions around the neutral axis. It is often desirable to define dead space between two other adjacent regions because the tilt sensor does not detect it, making it undetectable to the user, or for other reasons. If the neutral axis represents the Y-axis, angular deviations between approximately 91 degrees and -91 degrees (meaning an inverted device), or angular deviations of approximately 0 degrees, do not correspond to tilt regions. If control is selected when the device is not oriented towards a tilt region, the default output is output, the last output is output, no output is output, an output related to the nearest or complementary tilt region is output, or another type of output is output.

[0326] The angular deviation in region E404 originates from a strong negative gradient of the apparatus, while the angular deviation in region E405 also originates from a negative gradient smaller than that of region E404. The inclination deviation in region 407 originates from a strong positive gradient, while the angular deviation in region E406 also originates from a positive gradient smaller than that of region E407.

[0327] Figure 31C shows an example of two inclination regions defined around a neutral axis, where the region of approximately 0 degrees around the neutral axis is substantially the first region. In particular, if the neutral axis represents the X-axis, the device remains in region E409 if it is rotated negatively, does not move from the neutral position, or is rotated gently in the positive direction. A strong positive rotation is required for the device to be directed towards region E410. The inclination regions shown in Figure 31C are desirable, for example, where region E409 represents the default desired output, and aggressive, high-amplitude operation of the device is required to position this device in region E410, thereby disabling the default desired output. In the example in Figure 31C, inclination region E409 includes an angular deviation of 0 degrees, where the angular deviation of this device is within inclination region E409 if the angular deviation around the first axis is measured at 0 degrees.

[0328] Figure 31D shows an example of two inclination regions defined around the neutral axis, where a single region occupies bands of angular deviation on both sides of the neutral axis. More specifically, region E412 is defined by the region surrounding 0 degrees around the neutral axis, and region E411 occupies bands of symmetrical angles in the positive and negative angular directions. If the neutral axis represents the Z axis, the angular deviation in region E411 originates from high-amplitude positive or negative yaw. The angular deviation in region E412 originates from gentler positive or negative yaw, or from the orientation of the device in the neutral position.

[0329] In any of the examples above, the neutral axis may represent the X, Y, and / or Z axes, thus effectively increasing the total number of available inclination regions. For example, if the neutral axis in Figure 31A represents the X axis and the neutral axis in Figure 31B represents the Y axis, then a total of eight inclination regions are available, since the four gradient inclination regions in Figure 31B are each divided into the two rotational inclination regions of the example in Figure 31A. Assuming that each axis has an equal number n inclination regions, the total number of inclination regions for a two-axis configuration is n², and the total number of inclination regions for a three-axis configuration is n³.

[0330] Finally, in some examples, the angle deviation itself, rather than the tilt region, determines the output signal, so it is unnecessary to define the tilt region. Furthermore, the tilt region is also potentially defined if the range of motion around the desired axis is equally divided by the number of output signals (where each output signal corresponds to a mathematically determined range of angles).

[0331] Returning to Figure 30, the neutral position of the device is determined in relation to at least a first axis, and the device includes at least a first control related to a first plurality of output signals (step ES304).

[0332] Figure 32 shows an upper outer view of an example device relating to another exemplary embodiment. The device E500, i.e., a mobile phone, has a keypad including at least a first control E502 associated with a first set of output signals. In the example shown in the figure, the first control E502 is a key or button on the keypad or keyboard of the device E500, and each individual control represents a variety of alphanumeric characters or symbols. In particular, the first control E502 is labeled "9" and corresponds to four output signals representing the characters "W", "X", "Y", and "Z", or to twelve output signals representing the case-sensitive characters "W", "X", "Y", "Z", "w", "x", "y", and "z", as well as the symbols ",", ".", " / ", and "'". There is no limit to the number of output signals or characters that a single control can correspond to. In certain embodiments, the first control E502 is associated with multiple output signals, such as three output signals or nine output signals. Each of the multiple output signals may correspond to a character such as an alphanumeric character or a symbol.

[0333] The neutral position of the device E500 is determined, for example, prior to or after the selection of a first control, or when the device E500 is powered on at the location of operation. In one embodiment, a memory buffer stores the output data of the tilt sensor, and the neutral position of the device E500 is reconstructed from the orientation and output data of the device E500 when a control is selected. In another embodiment, the neutral position is a factory preset state, for example, defined as the neutral X-axis extending perpendicular to the center of the Earth, and the angular deviation is measured when the device E500 is facing any direction other than up. In a further embodiment, the processor, tilt sensor, and memory communicate to determine a common neutral position based on the average position of the device E500 whenever a control is selected as usual. In yet another embodiment, the neutral position is user selectable. In any respect, the neutral position effectively resets the tilt sensors to 0 degrees across each axis, and any movement of the device E500 away from the neutral position helps register the angular deviation. With respect to the user or the Earth, the neutral position of the device E500 is a flat, vertical, or oblique or inclined position.

[0334] In a further embodiment, the neutral position of the device E500 is determined perpendicular to the first axis with respect to at least a second axis, where the angular deviation includes the first axis component and the second axis component. In a further embodiment, the neutral position of the device E500 is determined perpendicular to the first axis and the second axis with respect to at least a third axis, where the angular deviation includes the third axis component. The first axis, the second axis, and / or the third axis intersect within the device E500, outside the device E500, along a peripheral position, or at the edge of the device E500.

[0335] Device E500 includes a tilt sensor that detects the orientation of the device, making text input to the device easy. For example, the tilt sensor detects the degree to which the device is rotated to the left, rotated to the right, or tilted up or down, where the deviation in the tilt direction or angle of the device around the axis indicates how the selection of control E502 is interpreted and output. For example, if control E502 corresponds to a number of characters, the orientation of device E502 identifies which of those characters was output when control E502 was selected, or identifies when the appropriate character was output.

[0336] By using the orientation of the device to identify the characters to be output, a character can be output each time a single control is selected, increasing the speed of text input by reducing the number of control selections required to input text. Since a fixed number of control selections represents character input, the user may identify the next character immediately after the current character is identified, eliminating the need to wait a predetermined number of times before identifying the next character, and further increasing the speed of text input.

[0337] As described above, the neutral position of the device is a reference direction in which the angular deviation is measured around at least one axis relative to the selected position, and this selected position corresponds to the position of the device at that time or after the selection of a control, such as a first control. In one embodiment, the neutral position of the device is determined in relation to one axis, which is determined as a "flat" position, where one axis is parallel to the ground. In another embodiment, the neutral position of the device is determined in relation to two axes, which is ergonomically determined as the orientation of the device when it is normally held by the user of the device. In a further embodiment, the neutral position of the device is determined in relation to three axes, where one axis is determined to be parallel to the magnetic north-south axis, one axis is determined to be parallel to the east-west axis, and the third axis is determined to be opposite to or away from the center of the Earth.

[0338] Returning to Figure 30, the angular deviation of the device is measured around at least the first axis (step ES305). In particular, a tilt sensor, such as the tilt sensor E206, measures the angular deviation between the device's current position and the neutral position, where the angular deviation includes components for each axis. In one embodiment, the tilt sensor E206 measures the angular deviation of the device when a control is selected. The selection of the control itself may affect the orientation of the device, and in another embodiment, the tilt sensor measures the angular deviation of the device before or after its control is selected.

[0339] A tilt sensor detects the orientation of a device. For example, a tilt sensor detects the degree to which the device is rotated left or right, tilted up or down, or swayed clockwise or counterclockwise. In one embodiment, the tilt sensor measures at least two separate levels of rotational tilt around the X-axis, in which case the device may be rotated left, rotated right, or neither left nor right. Furthermore, the tilt sensor measures at least two separate levels of gradient tilt around the Y-axis in the front-rear direction, in which case the device may be tilted up, tilted down, or neither up nor down. Furthermore, the tilt sensor measures at least two separate levels of sway tilt around the Z-axis, in which case the device may sway clockwise or counterclockwise, or not sway at all. In such an embodiment, if the device is rotated left by 1.5 to 4.5 degrees, the tilt sensor indicates that the device has rotated left. As another example, if the device is tilted forward by less than 1.5 degrees and backward by less than 1.5 degrees, the tilt sensor will indicate that the device is not tilted forward or backward. In another implementation, the tilt sensor may indicate four or more levels of tilt in each of the left-to-right and front-to-back directions. In such implementation, each level of tilt in a particular direction corresponds to a range of angles in which the device is tilted.

[0340] The angle deviation is displayed (step ES306). As described above, the direction of the neutral position does not need to be indicated to the user. Furthermore, each axis may have two or more tilt regions in each direction around each axis. For these and other reasons, the indicator is provided to display either the angle deviation or the corresponding tilt region in real time or near real time. If the angle deviation is measured before or after the control is selected, the indicator estimates the appropriate angle deviation or tilt region display at that time based on all available information. If the neutral position is defined in relation to two or more axes, the user can determine which axis the indicator will display, and the indicator may have such default or preset axis, or the determination may be context-dependent.

[0341] Figures 33A–33B illustrate an example of an indicator according to one exemplary embodiment. In Figure 33A, indicator E600 indicates the orientation of the device on the display. The indicator provides visual feedback so that the user is aware of the orientation of the device to use in interpreting control selections.

[0342] Indicator E600 includes a positive tilt indicator E601 and a negative tilt indicator E604, which point to the negative (left) and positive (right) directions, respectively. In addition, indicator E600 includes a center indicator E602, which is visually distinguishable from the positive tilt indicator E601 and the negative tilt indicator E604 when the device is not tilted, such as when the device is in a neutral position or a position not registered by the tilt sensor (e.g., upside down). When the device is tilted in the indicated direction, one of the tilt indicators is illuminated or otherwise visually distinguishable from the other tilt indicators and the center indicator E602. Furthermore, when the device is not rocked to the left or right, the center indicator E602 is illuminated or otherwise visually distinguishable from the positive tilt indicator E601 and the negative tilt indicator E604. The center indicator is illuminated, for example, when the device is oriented as shown in Figure 28. The positive tilt indicator E601 is illuminated when the device is oriented as shown in region E402 of Figure 31A. The negative tilt indicator E604 is illuminated when the device is oriented as shown in region E401 of Figure 31A.

[0343] In other embodiments illustrated in Figures 33B and 33C, indicator E605 also includes two partial tilt indicators E606 and E607, each pointing to a negative and positive direction, respectively. Each of the partial tilt indicators is positioned between the central indicator E604 and either the negative tilt indicator E604 or the positive tilt indicator E601. The partial tilt indicators are illuminated or otherwise visually distinguishable from the other components of indicator E605 when the device is partially tilted in the indicated direction. In one embodiment, both the partial tilt indicators and the central indicator are illuminated when the device is partially tilted in the corresponding direction. For example, when the device is oriented in the tilt region E404 of Figure 31B, the negative tilt indicator E604 is illuminated. When the device is oriented in the tilt region E405 of Figure 31B, the partial negative tilt indicator E606 and the central indicator E602 are illuminated. The central indicator 602 is illuminated when the device is oriented in the neutral position as shown in Figure 28. The partial positive tilt indicator E607 and the central indicator 602 are illuminated when the device is oriented in the tilt region E406 in Figure 31B. The positive tilt indicator E601 is illuminated when the device is oriented in the tilt region E407 in Figure 31B. Any number of tilt indicators or partial tilt indicators are considered for each axis. For axes having dozens of associated tilt regions, for example, the same number, more, or fewer tilt indicators may be used to provide visual feedback.

[0344] Figure 33D illustrates a two-axis tilt indicator that may be presented on a display. Although the axes referred to in relation to Figure 33D are called pitch (front-to-back) and roll (left-to-right), these designations are arbitrary, and a set of indicators can also be a yaw axis or other axes. Indicator E609 operates similarly to indicator E605 with respect to one axis. However, indicator E609 also integrates a pitch tilt indicator consisting of a negative pitch indicator E610, a partial negative pitch indicator E611, a partial positive pitch indicator E612, and a positive pitch indicator E614, in contrast to the previously described single-axis indicator E605 (which was described as a roll indicator). In other embodiments illustrated in Figure 33E, the indicator includes a single feature E615 indicating the importance of the orientation of the device. For example, a single feature indicator may output a number for measuring the angular displacement of the device.

[0345] Although the indicator is represented in Figures 28 and 33 as a series of arrows or intuitive illumination, in one embodiment the indicator is integrated into a display (e.g., display E105), or the indicator is a speaker that emits a sound or sound file to the user via audio that represents the tilt of the device. Furthermore, in another embodiment the angular displacement or tilt area is not displayed or does not occur.

[0346] Returning to Figure 30, the selection of the first control is received (step ES307). In one embodiment, the control is a keypad button, and the selection occurs when the user presses the button. This allows a signal to be generated and sent to the processor indicating that a keypad button selection has occurred. In another embodiment, the control is not a physical control but rather an icon on a touchscreen. In this embodiment, the selection occurs when the user touches the area of ​​the touchscreen associated with the icon. Here, the touchscreen application reads the coordinates of the touch, relates the coordinates to the position of the icon, and sends a signal indicating that the control has been selected. Other types of control selections are also considered.

[0347] According to the embodiment in Figure 32, device E500 includes a keypad, or grouping of controls, which allows the user to input text to interact with the GUI presented on display E505. Each control corresponds to a multiple output signal (each output signal associated with a character). In one embodiment, the keypad includes eight controls numbered "2" through "9," each corresponding to multiple characters and numbers. For example, the control numbered "2" corresponds to the characters "A," "B," and "C." In addition, other controls included in the keypad perform other text input functions. For example, the control numbered "*" is used to change the case of the next character to be output. The control numbered "0" is used to advance to the next character after the current character has been characterized, and the control numbered "#" is used to insert the character "space."

[0348] One of the first set of multiple output signals is output based on at least selection and angular displacement (step ES309), or at least selection, angular displacement and multiple tilt regions. Since the first control is associated with the first set of multiple output signals, the angular displacement, or the angular displacement and multiple tilts, is used to determine which one of the first set of multiple output signals is output. In one embodiment, the neutral position of the device is determined with respect to one axis, where three tilt regions are defined around that one axis, and the first control is associated with the three tilt regions. In this case, if the angular displacement is in the first tilt region, the first output signal is output. If the angular displacement is in the second tilt region, the second output signal is output, and if the angular displacement is in the third tilt region, the third output signal is output. In another embodiment, the output signals are output based on the angular displacement and a numerical value of the output signal associated with the first control based on a formula or algorithm.

[0349] Various figures show front and side views of the device in Figure 32 in different operating states. In particular, Figures 34A and 34B illustrate the front and side views of device E500 in the neutral position, respectively. Figure 35A shows a front view of the device when operated in a negative roll around the X-axis. Figure 35B shows a front view of the device when operated in a positive roll around the X-axis. Similarly, Figure 36A shows a side view of the device when operated in a positive pitch around the Y-axis. Figure 36B shows a side view of the device when operated in a negative pitch around the Y-axis. In Figures 35 and 36, the device is tilted approximately ±30 degrees around each axis from the neutral position shown in Figure 34.

[0350] The device's orientation, as indicated by the angular displacement measured by the tilt sensor, affects the output signals the device outputs when a keypad control is selected, for example, influencing the character generated by the control selection. Each of the multiple characters or output signals represented by a single keypad control corresponds to a different orientation of the device. When one of the keypad controls is selected, the device identifies the multiple characters corresponding to the selected control, and the orientation of the device as indicated by the tilt sensor. One of the multiple characters and the case for the character are identified based on the identified orientation, and the identified character is output.

[0351] The degree to which the device is rolled left or right when a control is selected affects which of the multiple characters represented by the control is output. In one embodiment, a control representing multiple characters represents three characters, and the characters represented by the control are arranged left to right on the control. The device is configured to indicate whether the device is rolled left, rolled right, or not rolled left or right. In such an embodiment, rolling the device to the left when the control is selected indicates that the leftmost character should be output. Similarly, rolling the device to the right when the control is selected indicates that the rightmost character should be output. Finally, holding the device in a neutral position when the control is selected indicates that the middle character should be output.

[0352] In other embodiments, rolling the device to the left when a control is selected indicates that the rightmost character should be output. Rolling the device to the right when a control is selected indicates that the leftmost character should be output. Holding the device in the neutral position when a control is selected indicates that the center character should be output. For example, this type of embodiment may be used because rolling the device to the left makes the rightmost character appear higher and more prominent than the other characters, and rolling the device to the right makes the leftmost character appear higher and more prominent than the other characters.

[0353] In other embodiments, keypad controls represent more than three characters (e.g., three letters and a number, or four letters and a number). For example, a control with a "7" on a conventional phone corresponds to the letters "P", "Q", "R", and "S", as well as the number "7". In this type of case, the tilt sensor is configured to identify three or more separate left and right roll positions so that one of the three or more characters represented by the selected control can be identified based solely on the device's roll direction. Each of the separate roll positions corresponds to one of the characters represented by the selected control. For example, if the selected control is a key with a "7", the device rolling as shown in area E404 of Figure 31B indicates that the letter "P" should be output. The device rolling as shown in area E405 of Figure 31B indicates that the letter "Q" should be output. The device rolling as shown in area E406 of Figure 31B indicates that the letter "R" should be output. The device rolling as shown in area E407 of Figure 31B indicates that the letter "S" should be output. Furthermore, as illustrated in Figure 28, a device whose orientation is set to the neutral position should output the number "7".

[0354] The device's roll direction is used to identify the character being output, while the device's pitch direction is used to identify the case for that character. In one embodiment, a device that is pitched (or tilted) forward when a control is selected will output the character identified by the device's roll (left-right tilt) direction in uppercase. Similarly, a device that is not pitched forward or backward (in a neutral pitch position) when a control is selected will output the character identified by the device's roll (left-right tilt) direction in lowercase.

[0355] In some embodiments, a device that is pitched (or tilted) backward may output a symbol. This symbol may be a symbol corresponding to a number represented by a selected control on a conventional computer keyboard. For example, if the control representing the number "1" is selected when the device is tilted backward, the symbol "!" may be output because the symbol "!" corresponds to the number "1" on a conventional computer keyboard (for example, pressing "Shift" and "1" on a computer keyboard outputs the character "!").

[0356] The tilt sensor has the ability to detect tilt positions in more pitch directions than are necessary to indicate the case of the output character. Thus, pitch positions not used to indicate the case of a character may be used to select a character. For example, the control may represent three characters and a number, and three roll positions may be used to select from among the three characters. Two pitch positions may select the case for the character, and a third pitch tilt position may select a number represented by a key.

[0357] Furthermore, the tilt sensor independently indicates whether the device is rolled to the left, neutral, or right, or whether the device is pitched forward, neutral, or backward. This allows the tilt sensor to indicate whether the device is in one of nine directions. Each of the nine directions may correspond to a letter and a case for that letter.

[0358] Figure 37 is a table showing one possible mapping of device orientation to output signals corresponding to the characters and cases that may be output when the control marked "2" on the keypad is selected. In the illustrated mapping, a device rolled to the left and pitched forward will output the uppercase letter "A". A device that is neither rolled nor pitched in any direction will output the lowercase letter "b". And a device pitched backward will output the number "2". In other embodiments where the tilt sensor can identify three or more roll positions or three or more pitch positions, more orientations that can be positioned on the characters and cases are available.

[0359] The output signal corresponding to a character is described as being selected based on the angular displacement or tilt position of the device's first axis. The output signal corresponding to uppercase or lowercase letters is described as being selected based on the angular displacement or position of the device's second axis. In other embodiments, angular displacements of separate axes may achieve the output of the signal corresponding to the character or uppercase or lowercase letter. In general, any orientation of the device may be positioned for any character and / or case for any character, regardless of which axis is used to select the character or case.

[0360] In addition to outputting signals corresponding to characters output in response to control selections, the orientation of the device may be used to indicate menu options to be selected. For example, selecting a control that does not correspond to any character (e.g., the "1" key on a telephone) causes the telephone's display to present a menu (each option in the menu corresponding to a different orientation on the telephone). When a control indicating that a selection must be made from a menu (e.g., the "OK" key, the "Enter" key, or the "1" key) is selected, the orientation of the device may indicate which of the menu options is selected. In one embodiment, when the "1" key is selected, a menu of symbols similar to those illustrated in Figures 38A and 38B is displayed. Tilting the device to select the "1" key in the same way as before may cause the corresponding symbol to be output. After the symbol has been output, characters and numbers may be output as described above until the "1" key is selected again to display the symbol menu in the same way as before. Turning the device completely upside down, shaking the device, or moving the device in a way that is not interpreted as tilting the device generates other menus.

[0361] If the angular displacement is within the first inclination region when the selection is received, a first output signal is output. On the other hand, if the angular displacement is within the second inclination region when the selection is received, a second output signal is output. Furthermore, if the angular displacement is within the third or fourth inclination region when the selection is received, a third or fourth output signal is output, respectively.

[0362] If multiple inclination regions of a first axis are defined around the first axis, and multiple inclination regions of a second axis are defined around the second axis, then one of the first multiple output signals may be output based on the multiple inclination regions of the first axis and / or the multiple inclination regions of the second axis. When a selection is received, if the component of the first axis is in the first inclination region of the first axis and the component of the second axis is in the first inclination region of the second axis, then the first output signal may be output. If the component of the first axis is in the second inclination region of the first axis and the component of the second axis is in the first inclination region of the second axis, then the second output signal may be output. If the component of the first axis is in the second inclination region of the first axis and the component of the second axis is in the second inclination region of the second axis, then the third output signal may be output. And / or, a fourth output signal may be output if the component of the first axis is in the slope region of the second first axis and the component of the second axis is in the slope region of the second second axis.

[0363] Alternatively, in another embodiment, a first output signal may be output if, when the selection is received, the first component is in the tilt region of the first axis and the component of the second axis is in the tilt region of the first axis. A second output signal may be output if the first component is in the tilt region of the first axis and the component of the second axis is in the tilt region of the second axis. A third output signal may be output if the first component is in the tilt region of the first axis and the component of the second axis is in the tilt region of the third axis. A fourth output signal may be output if the first component is in the tilt region of the second axis and the component of the second axis is in the tilt region of the first axis. A fifth output signal may be output if the first component is in the tilt region of the second axis and the component of the second axis is in the tilt region of the second axis. A sixth output signal may be output if the first component lies within the inclination region of the second first axis and the second axis component lies within the inclination region of the third second axis. A seventh output signal may be output if the first component lies within the inclination region of the third first axis and the second axis component lies within the inclination region of the first second axis. An eighth output signal may be output if the first component lies within the inclination region of the third first axis and the second axis component lies within the inclination region of the second second axis. And / or, a ninth output signal may be output if the first component lies within the inclination region of the third first axis and the second axis component lies within the inclination region of the third second axis.

[0364] The output signal is displayed (step ES310), and method E300 ends (step ES311). The output signal is displayed on a display such as display E105. In an alternative embodiment, the output signal is not displayed.

[0365] In the embodiment shown in Figure 32, device E500 also includes display E505, which is used to present a graphical user interface ("GUI") to the user of device E500. The GUI allows the user of device E500 to perform functions that prompt the user to enter text into device E500. For example, the user may identify an entry for a person within the scope of the phonebook stored in device E500 by entering the person's name. As another example, the user may add an entry for a person to the phonebook by entering information describing the person (e.g., the person's name and one or more phone numbers used by that person). Furthermore, the GUI allows the user to identify a text message to be sent from device E500 or other text memos to be stored in device E500. Device E500 also displays a GUI that allows the user to identify a text message.

[0366] Interpreting control selections based on device orientation increases the number of operations that can be performed by a single control selection. For example, each control selection may be interpreted in many ways equal to the number of different device orientations that may be detected. Furthermore, the device orientation may indicate how a control selection that does not correspond to any character may be interpreted. Thus, a user may be able to quickly perform relatively complex operations simply by tilting the device to select a control. For example, selecting the "*" key while the device is rolled to the left may result in a specific mode of text input (e.g., numbers only, all uppercase) used for text input until the next opportunity to select the "*" key while the device is rolled to the left. In another embodiment, the tilt sensor achieves tilt scrolling. In this case, upon receiving a control selection, the user interface scrolls in accordance with the direction of the tilt. For example, a forward pitch that occurs when a control is selected results in scrolling the user interface or menu items on the user interface upwards.

[0367] In other general embodiments, a computer program product, obviously stored on a computer-readable medium, is described in detail. The computer program product is operable to cause a computer to perform operations including determining a neutral position of a device, which includes a first control associated with at least a plurality of first output signals, with respect to at least a first axis, and measuring the angular displacement of the device, at least around the first axis. The computer program product is also operable to cause a computer to perform operations including receiving a selection of the first control and outputting at least one of a plurality of first output signals based on the selection and angular displacement.

[0368] Finally, although many embodiments have been described or illustrated as telephone devices, the concepts relating herein are by no means limited to telephones and are considered to be applicable in fact to a wide variety of devices, including any device where the number of controls is minimized for device design and layout constraints. Sample devices include computer keyboards, remote control devices, watches, joysticks or game controllers, or other computer input or consumer electronic devices.

[0369] Therefore, many embodiments have been described. Nevertheless, it is understood that various modifications may be made. For example, elements of separate embodiments may be combined, supplemented, or removed to create other embodiments. Furthermore, various techniques may be used, combined, and modified to create embodiments. Such techniques include, for example, various digital electronic circuits, hardware, software, firmware, integrated components, discrete components, processing devices, memory storage devices, communication devices, lenses, filters, display devices, and projection devices.

[0370] (Game System) With reference to Figure 39, several embodiments of the game system 39 are described. Figure 39 is an external view illustrating the game system 39. In the following description, several embodiments of the game system 39 include a fixed game device.

[0371] As shown in Figure 39, the game system F1 includes a fixed game device (hereinafter simply referred to as "game device") F3. It is connected via a connecting cord to a display (hereinafter referred to as "monitor") F2, such as a home television receiver with a speaker F2a, and to a controller F7 for providing operation information to the game device F3. The game device F3 is connected to a receiving unit F6 via a connecting terminal. The receiving unit F6 receives transmission data wirelessly transmitted from the controller F7. The controller F7 and the game device F3 are interconnected by wireless communication. The game device F3 is detachably mounted on an optical disc F4, as an example of a replaceable information storage medium. The game device F3 includes a power on / off switch, a game / process reset switch, and an open switch for opening the top lid of the game device F3 on its top main surface. When the player presses the open switch, the lid is opened. As a result, the optical disc F4 can be mounted or removed.

[0372] Furthermore, the game device F3 is equipped with an external memory card F5 that can be attached and detached as needed. The external memory card F5 has built-in backup memory for permanently storing saved data. The game device F3 executes game programs stored on the optical disc F4 and displays the results as game images on the monitor F2. The game device F3 can also use the saved data stored on the external memory card F5 to replay the state of previously played games and display the game images on the monitor F2. A player playing with the game device F3 can enjoy the game by operating the controller F7 while viewing the game images displayed on the monitor F2.

[0373] Controller F7 wirelessly transmits data from its internally contained communication unit F75 (described later) to the game device F3 connected to the receiving unit F6, for example, using Bluetooth® technology. Controller F7 has two control units, a core unit F70 and a sub-unit F76, which are interconnected by a flexible connection cable F79. Controller F7 is primarily an operating means for manipulating player objects appearing in the game space displayed on monitor F2. The core unit F70 and sub-unit F76 each include an operation unit (e.g., multiple operation buttons, keys, sticks, etc.). As will be described in detail later, the core unit F70 includes an imaging information calculation unit F74 for taking images seen from the core unit F70. As an embodiment of the imaging target of the imaging information calculation unit F74, two LED modules F8L and F8R are provided near the display screen of monitor F2. LED modules F8L and F8R each emit infrared light forward from monitor F2. In this embodiment, although the core unit F70 and the sub-unit F76 are interconnected by a flexible cable, the sub-unit F76 may have a wireless unit, thereby eliminating the connecting cable F79. For example, the sub-unit F76 may have a Bluetooth® unit as its wireless unit. This allows the sub-unit F76 to transmit operation data to the core unit F70.

[0374] Next, the structure of the game device F3 is described with reference to Figure 40. Figure 40 is a functional block diagram of the game device F3.

[0375] As shown in Figure 40, the game device F3 includes, for example, a RISC CPU (Central Processing Unit) F30 for executing various types of programs. The CPU F30 executes a boot program stored in a boot ROM (not shown) to initialize the main memory, including the main memory F33, and then executes a game program stored in an optical disc F4 to perform the game process according to the game program. The CPU F30 is connected to a GPU (Graphics Processing Unit) F32, main memory F33, DSP (Digital Signal Processor) F34, and ARAM (Audio RAM) F35 via a memory controller F31. The memory controller F31 is connected to a controller I / F (Interface) F36, video I / F F37, external memory I / F F38, audio I / F F39, and disk I / F F41 via a predetermined bus. The controller interface F36, video interface F37, external memory interface F38, audio interface F39, and disk interface F41 are connected to the receiving unit F6, monitor F2, external memory card F5, speaker F2a, and disk drive F40, respectively.

[0376] The GPU F32 performs image processing based on instructions from the CPU F30. The GPU F32 includes, for example, semiconductor chips for performing the computational processes necessary to display 3D images. The GPU F32 performs image processing using memory dedicated to image processing (not shown) and a portion of the storage area of ​​the main memory F33. Using this type of memory, the GPU F32 generates game image data and movies to be displayed on monitor F2 and, if necessary, outputs the generated data or movies to monitor F2 via the memory controller F31 and video I / F F37.

[0377] Main memory F33 is a memory area used by CPU F30, and stores game programs and other data necessary for processing performed by CPU F30 as needed. For example, main memory F33 stores game programs and other data of various types read from optical disc F4 by CPU F30. Game programs and other data of various types stored in main memory F33 are executed by CPU F30.

[0378] The DSP F34 processes sound data and other information generated by the CPU F30 during the execution of the game program. The DSP F34 is connected to the ARAM F35, which stores sound data and other information. The ARAM F35 is used when the DSP F34 executes a predetermined process (for example, the game program or storing already loaded sound data). The DSP F34 reads the sound data stored in the ARAM F35 and outputs that sound data to the speaker F2a included in the monitor F2 via the memory controller F31 and audio interface F39.

[0379] The memory controller F31 comprehensively controls data transmission and is connected to the various I / Fs described above. The controller I / F F36 includes, for example, four controller I / Fs F36a, F36b, F36c, and F36d, and transmits the game device F3 to external devices that can engage via the connectors of the controller I / Fs F36a, F36b, F36c, and F36d. For example, the receiving unit F6 engages with this type of connector and is connected to the game device F3 via the controller I / F F36. As described above, the receiving unit F6 receives the transmission data from the controller F7 and outputs the transmission data to the CPU F30 via the controller I / F F36. The video I / F F37 is connected to the monitor F2. The external memory I / F F38 is connected to the external memory card F5 and can access backup memory and the like provided on the external memory card F5. The audio interface F39 is connected to the speaker F2a built into the monitor F2 so that sound data read from the ARAM F35 by the DSP F34 or sound data output directly from the disk drive F40 can be output from speaker F2a. The disk interface F41 is connected to the disk drive F40. The disk drive F40 reads the data stored at a predetermined read position on the optical disk F4 and outputs that data to the bus of the game device F3 or to the audio interface F39.

[0380] Next, the controller F7 is described with reference to Figures 41 and 42. Figure 41 is a perspective view illustrating the external appearance of the controller F7. Figure 42 is a perspective view illustrating the state of the connection cable F79 of the controller F7 shown in Figure 41, which is connected to or disconnected from the core unit F70.

[0381] As shown in Figure 41, the controller F7 includes a core unit F70 and a subunit F76 that are interconnected by a connecting cable F79. The core unit F70 has a housing F71 that includes a plurality of operation units F72. The subunit F73 has a housing F77 that includes a plurality of operation units F78. The core unit F70 and the subunit F76 are interconnected by a connecting cable F79.

[0382] As shown in Figure 42, the connecting cable F79 has a connector F791 at one end that is detachably connected to the connector F73 of the core unit F70. The other end of the connecting cable F79 is fixedly connected to the subunit F76. The connector F791 of the connecting cable F79 engages with the connector F73 provided on the rear surface of the core unit F70 in order to connect the core unit F70 and the subunit F76 to each other by the connecting cable F79.

[0383] The core unit F70 is described with reference to Figures 43 and 44. Figure 43 is a perspective view of the core unit F70 from above and from the rear. Figure 44 is a perspective view of the core unit F70 from below and from the front.

[0384] As shown in Figures 43 and 44, the core unit F70 includes a housing F71 formed by plastic molding or the like. The housing F71 has a generally parallelepiped shape that extends longitudinally from front to back. The overall size of the housing F71 is small enough to be held in one hand, even by an adult or a child.

[0385] A directional pad F72a is located in the center of the front side of the top surface of housing F71. The directional pad F72a is a cross-shaped four-way push switch. The directional pad F72a includes operating parts corresponding to four directions (forward, backward, right, and left) represented by arrows. These are positioned on cross-shaped protrusions arranged at 90-degree intervals. The player selects one of the forward, backward, right, or left directions by pressing one of the operating parts of the directional pad F72a. Through the operation of the directional pad F72a, the player can, for example, indicate the direction in which the player's character appearing in the virtual game world moves, or the direction in which the cursor moves.

[0386] Although the directional pad F72a is an operation unit for outputting operation signals in accordance with the directional input operations performed by the player as described above, this type of operation unit may be provided in other forms. For example, the directional pad F72a may be replaced with a composite switch including a push switch with an annular four-way operation unit and a center switch located at its center. Alternatively, the directional pad F72a may be replaced with an operation unit including a tiltable stick protruding from the top surface of the housing F71 and outputting operation signals in accordance with the tilt direction of the stick. Alternatively, the directional pad F72a may be replaced with an operation unit including a horizontally slidable disc-shaped member and outputting operation signals in accordance with the sliding direction of the disc-shaped member. Alternatively, the directional pad F72a may be replaced with a touchpad. Alternatively, the directional pad F72a may be replaced with an operation unit including switches representing at least four directions (forward, backward, right, left) and outputting operation signals in accordance with the switches pressed by the player.

[0387] Behind the directional pad F72a on the top surface of the housing F71 are several operation buttons F72b, F72c, F72d, F72e, F72f, and F72g. Each of the operation buttons F72b, F72c, F72d, F72e, F72f, and F72g is an operation unit that outputs the respective operation signals assigned to the operation buttons F72b, F72c, F72d, F72e, F72f, and F72g when the player presses its head. For example, operation buttons F72b, F72c, and F72d are assigned the functions of the first button, the second button, and the A button. Furthermore, for example, operation buttons F72e, F72f, and F72g are assigned the functions of the minus button, the home button, and the plus button. The operation buttons F72b, F72c, F72d, F72e, F72f, and F72g are each assigned a function according to the game program executed by the game device F3. In the exemplary device shown in Figure 43, the operation buttons F72b, F72c, and F72d are located on the center line in the front-to-back direction on the top surface of the housing F71. The operation buttons F72e, F72f, and F72g are located on the left-to-right line between the operation buttons F72b and F72d on the top surface of the housing F71. The top surface of the operation button F72f is embedded within the top surface of the housing F71 to prevent accidental pressing by the player.

[0388] An operation button F72h is located in front of the directional pad F72a on the top surface of housing F71. Operation button F72h is a power switch for remotely turning the power of the game device 3 on or off. To prevent accidental pressing by the player, the top surface of operation button F72h is embedded within the top surface of housing F71.

[0389] Multiple LEDs F702 are provided behind the operation button F72c on the top surface of the housing F71. Controllers F7 are assigned a controller type (number) to distinguish them from other controllers F7. For example, LED F702 is used to inform the player of the controller type currently set for the controller F7 that he or she is using. Specifically, when the core unit F70 transmits data to the receiving unit F6, one of the multiple LEDs F702 corresponding to the controller type is illuminated.

[0390] On the upper surface of the housing F71, a sound hole for outputting sound from the speaker F706 shown in Figure 45 is provided between the operation buttons F72e, F72f, and F72g and the operation button F72b, as described below.

[0391] A concave portion is formed on the bottom surface of the housing F71. As will be described in detail later, the concave portion is formed at a position where the player's index finger or middle finger is positioned when the player holds the core unit F70. An operation button F72i is provided on the rearward sloping surface of the concave portion. The operation button F72i is an operation unit that functions, for example, as a B button. The operation button F72i is used, for example, as a trigger switch in a shooting game or to draw the player's attention to a predetermined target.

[0392] An image pickup element F743, included in the image information calculation unit F74, is provided on the front surface of the housing F71. The image information calculation unit F74 is a system for analyzing image data taken by the core unit F70 and detecting the centroid, size, etc., of areas with high brightness in the image data. The image information calculation unit F74 has a maximum sampling interval of, for example, approximately 200 frames per second, and therefore can track and analyze even relatively fast movements of the core unit F70. The image information calculation unit F74 will be described in detail later. A connector F73 is provided on the rear surface of the housing F71. The connector F73 is, for example, a 32-pin edge connector and is used to engage and connect the core unit F70 with the connector F791 of the connecting cable F79.

[0393] The internal structure of the core unit F70 is described with reference to Figures 45 and 46. Figure 45 is a perspective view of the core unit F70 from the rear, showing the upper casing (part of the housing F71) of the core unit F70 removed. Figure 46 is a perspective view of the core unit F70 from the front, showing the lower casing (part of the housing F71) of the core unit F70 removed. Figure 46 is a perspective view showing the back side of the substrate F700 shown in Figure 45.

[0394] As shown in Figure 45, the substrate F700 is fixed inside the housing F71. Operation buttons F72a, F72b, F72c, F72d, F72e, F72f, F72g, and F72h, an acceleration sensor F701, an LED F702, an antenna F754, and the like are provided on the upper main surface of the substrate F700. These elements are connected to a microcomputer F751 (see Figures 46 and 55) and the like via lines (not shown) formed on the substrate F700 and the like. With a wireless module F753 (see Figure 55) and an antenna F754 (not shown), the core unit F70 can function as a wireless controller. A crystal oscillator F703 (not shown) provided in the housing F71 generates a reference clock for the microcomputer F751, which will be described later. A speaker F706 and an amplifier F708 are provided on the upper main surface of the substrate F700. The acceleration sensor F701 is located near the edge of the substrate F700, offset from its center. Therefore, acceleration, including changes in the direction of gravitational acceleration and centrifugal force components, can be detected based on the rotation of the core unit F70 around its longitudinal direction. As a result, a predetermined calculation is used to determine the rotation of the core unit F70 with desirable accuracy based on the detected acceleration data.

[0395] As shown in Figure 46, an image information calculation unit F74 is provided on the front edge of the bottom main surface of the substrate F700. The image information calculation unit F74 includes an infrared filter F741, a lens F742, an image pickup element F743, and an image processing circuit F744, all located in this order from the front surface of the core unit F70 on the bottom main surface of the substrate F700. A connector F73 is attached to the rear edge of the bottom main surface of the substrate F700. Furthermore, a sound IC F707 and a microcomputer F751 are provided on the bottom main surface of the substrate F700. The sound IC F707, which is connected to the microcomputer F751 and amplifier F708 via lines formed on the substrate F700, outputs an audio signal to the speaker F706 via the amplifier F708 based on sound data transmitted from the game device F3. A vibrator F704 is provided on the bottom main surface of the substrate F700. The vibrator F704 is, for example, a vibration motor or solenoid. The core unit F70 vibrates due to the operation of the vibrator F704, and this vibration is transmitted to the player's hand holding the core unit F70. Thus, a so-called vibration feedback game is realized. The vibrator F704 is positioned slightly forward of the housing F71. This allows the housing F71 held by the player to vibrate strongly, and the player can easily sense this vibration.

[0396] Subunit F76 is described with reference to Figures 47-50. Figure 47 is a perspective view illustrating a first embodiment of subunit F76. Figure 48 is a perspective view illustrating subunit F76 with the upper casing (part of housing F77) removed, as shown in Figure 47. Figure 49A is a top view illustrating a second embodiment of subunit F76. Figure 49B is a bottom view illustrating a second embodiment of subunit F76. Figure 49C is a left side view illustrating a second embodiment of subunit F76. Figure 50 is a perspective view of the second embodiment of subunit F76, viewed from the upper front side.

[0397] As shown in Figure 47, the subunit F76 includes a housing F77 formed, for example, by plastic mol...

Claims

1. A device including at least one processor, The aforementioned at least one processor is The transmitter of the device is configured to control the transmission of a signal to an electronic device located outside the device that encodes an identifier unique to the device, indicates the current location of the device, and causes the electronic device to display a message. In response to the transmission, the electronic device authenticates its unique identifier and determines that its current location is within a predetermined area, and thereafter, at least one processor transmits a first control signal to the electronic device for controlling an application in the electronic device, and the electronic device is capable of receiving a second control signal from a plurality of other devices outside of the device, and the electronic device responds to at least one of the first and second control signals, The aforementioned at least one processor further, Receiving a second signal from the motion sensor of the device indicating at least one movement of the device, Converting the second signal into a third control signal for controlling the application in the electronic device, The transmitter transmits the third control signal to the electronic device. The receiver of the device receives commands wirelessly received by the receiver from the electronic device, and It is configured to control the provision of information regarding the third control signal to the output device of the equipment in response to the command, The information provides guidance on specific actions that the user of the device should perform in a game played by the application, and the guidance includes instructions to inform the user of a tilt orientation, which indicates how to rotate the device itself with respect to at least one of the following axes: the longitudinal axis of the device, the width axis of the device perpendicular to the longitudinal axis, and the thickness axis of the device perpendicular to the longitudinal axis and the width axis. device.

2. The switch further includes two stable positions, The aforementioned at least one processor is The position of the switch is detected, and The apparatus according to claim 1, configured to control the transmitter transmitting a signal only when the switch is in the first of the two stable positions.

3. The apparatus according to claim 1, wherein the apparatus includes a wristwatch.

4. The device according to claim 1, wherein the device includes a wristband.

5. The apparatus according to claim 1, wherein the application includes a game application.

6. The apparatus according to claim 1, wherein the output device includes an audio speaker for providing the information to the user's player.

Images