Device operation control

Abstract

To address problems of conventional techniques.SOLUTION: A device includes: a display that displays information in a plurality of independent display regions each being independently identifiable; a memory that stores motion type control data; identifications of motion types and corresponding operations to be performed on information displayed in the display regions; a motion detecting arrangement that detects motion of the device; a user input arrangement that receive a user input identifying operations to be performed on the information displayed in the display regions during a learning phase; and a processor. The processor identifies, when the user input is received in the learning phase, the motion type from the motion detected by the motion detecting arrangement, and stores the received user input. Subsequent to the learning phase, when the motion detecting arrangement detects the motion, using the motion type control data, the processor identifies the motion type, and executes the corresponding operation on the information displayed in the display regions.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a device and a method for controlling the device using detected motion types. [Background technology]

[0002] Portable and moveable devices, such as mobile phones, tablet computing devices, and laptops, are in widespread use. A challenge for designers of portable devices is to design equipment that is lightweight, battery efficient, and convenient to use. One aspect of ease of use is the ease with which a user can input control commands. Summary of the Invention

[0003] The present invention provides a device including a display that displays information in a plurality of independent display areas, each display area being independently identifiable; a memory that stores action type control data including an identification of an action type and a corresponding operation to be performed on the information displayed in one or more identified display areas; an action detection arrangement that detects actions of the device; a user input arrangement that receives user input identifying an operation to be performed on the information displayed in the one or more identified display areas during a learning phase; and a processor that is programmed to: during the learning phase, when a user input is received that identifies an operation to be performed on the information displayed in the one or more identified display areas, identify an action type from the actions detected by the action detection arrangement, store the received user input corresponding to the data identifying the action type in the action type control data, and following the learning phase, when the action detection arrangement detects an action, use the action type control data to identify the action type and perform the corresponding operation on the information displayed in the one or more identified display areas.

[0004] The present invention also provides a method for controlling the display of information in multiple independent display areas on a device having a motion detection arrangement that detects motion of the device, each display area being independently identifiable, the method comprising: during a learning phase, receiving user input identifying an operation to be performed on information displayed in one or more identified display areas; detecting motion of the device using the motion detection arrangement; identifying an action type from the motion detected by the motion detection arrangement; storing the received user input corresponding to the data identifying the action type as action type control data; and repeating the receiving, detecting, identifying, and storing; and in an operation phase, detecting motion of the device using the motion detection arrangement and identifying the action type using the action type control data and performing the corresponding operation on the information displayed in the one or more identified display areas.

[0005] The present invention also provides a device including a memory storing control data including identification of an action type, a corresponding control operation, and corresponding gesture type data; an action detection arrangement for detecting actions of the device; a gesture detection arrangement for detecting gestures; a user input arrangement for receiving user input identifying the control operation during a learning phase; and a processor, the processor being programmed to: during the learning phase, when the user input identifying the control operation is received, identify an action type from actions detected by the action detection arrangement, store the received user input corresponding to the data identifying the action type in the control data, and determine corresponding gesture type data; and, following the learning phase, when the action detection arrangement detects an action, use the control data to identify the action type and perform the corresponding control operation, or when the gesture detection arrangement detects a gesture, use the control data to identify the gesture type and perform the corresponding control operation. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary handheld device.

[0007] [Figure 2] FIG. 10 is a schematic diagram illustrating an alternative display of an exemplary device.

[0008] [Figure 3] FIG. 10 is a schematic diagram illustrating an alternative display of an exemplary device.

[0009] [Figure 4] FIG. 2 is a schematic diagram illustrating the electronic components of an exemplary device.

[0010] [Figure 5] FIG. 1 is a flow diagram illustrating an exemplary learning phase.

[0011] [Figure 6] FIG. 1 is a flow diagram illustrating exemplary operational phases. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description and which show, by way of illustration, specific embodiments in which the present subject matter may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice them. It is to be understood that other embodiments may be utilized and that structural, logical, and electrical changes may be made without departing from the scope of the inventive subject matter. Such embodiments of the inventive subject matter may be individually and / or collectively referred to herein by the term "invention" merely for convenience and without any intention to intentionally limit the scope of this application to any single invention or inventive concept if more than one is actually disclosed.

[0013] Therefore, the following description is not to be taken in a limiting sense, and the scope of the inventive subject matter is defined by the appended claims and their equivalents.

[0014] In the following embodiments, equivalent components are marked with equivalent reference numbers.

[0015] In the following embodiments, the terms data store or memory are intended to encompass any computer-readable storage medium and / or device (or collection of data storage media and / or devices). Examples of data stores include, but are not limited to, optical disks (e.g., CD-ROMs, DVD-ROMs, etc.), magnetic disks (e.g., hard disks, floppy disks, etc.), and memory circuits (e.g., solid-state drives, random access memories (RAMs), etc.).

[0016] In one embodiment, the functions or algorithms described herein are implemented in hardware, software, or a combination of software and hardware. Software includes computer-executable instructions stored on a computer-readable carrier medium, such as a memory or other type of storage device. Furthermore, the described functions may correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the described embodiment is merely an example. Software may be executed on a digital signal processor, an ASIC, a microprocessor, or other type of processor.

[0017] Some embodiments implement the functionality in two or more specific interconnected hardware modules or devices, or as part of an application specific integrated circuit, with relevant control and data signals communicated between and through the modules. Thus, the exemplary process flows are applicable to software, firmware, and hardware implementations.

[0018] A generalized embodiment provides a device including: a display that displays information in a plurality of independent display regions, each display region being independently identifiable; a memory that stores motion type control data including identifications of motion types and corresponding operations to be performed on information displayed in one or more identified display regions; a motion detection arrangement that detects motion of the device; a user input arrangement that receives user input identifying operations to be performed on information displayed in the one or more identified display regions during a learning phase; and the user input arrangement that identifies operations to be performed on information displayed in the one or more identified display regions; and a processor that is programmed to: during the learning phase, when user input identifying an operation to be performed on information displayed in the one or more identified display regions is received, identify a motion type from the motion detected by the motion detection arrangement and store the received user input corresponding to data identifying the motion type in the motion type control data; and following the learning phase, when the motion detection arrangement detects a motion, identify the motion type using the motion type control data and perform the corresponding operation on the information displayed in the one or more identified display regions.

[0019] A user of the device can train the device on preferred action types to be used as inputs that cause operations to be performed on information displayed in one or more identified display areas. In this way, the user can tailor how the user interacts with particular identified display areas of the device to select preferred action types for particular operations to be performed on information displayed in one or more identified display areas. Identification of action types can include recorded action data for each action type for matching with detected action data to identify the action type from the action data. In one embodiment, not all of the action types need to be learned; some can be pre-stored.

[0020] The present invention is suitable for, but is not limited to, portable devices. The device can be any device that can be moved in a reciprocal, rotational, or shaking motion. Such devices are typically portable and wireless, but can simply be movable and connected by a wire. The device can be part of another apparatus and therefore detachable during use to allow the shaking, rotational, or reciprocating motion to be effected by the user to control the device. The device can include a mobile phone, a tablet computer, a wearable device such as a smart watch or personal monitor device, a laptop, or a machine control device.

[0021] The operation to be performed on the information displayed in one or more identified display regions may include any operation on the information in the display regions, such as cutting, pasting, inserting, or deleting displayed information in one or more display regions of the display. The operation may be limited to operating on one display region, a subset of the display regions, or all display regions.

[0022] The motion types that a user can use to identify and execute a control operation can include, for example, any of linear motion, reciprocating motion, rotational motion, swaying motion, see-sawing motion, swinging motion, or rocking motion. The displacement or amplitude of the "shake," the frequency or rate of the "shake," and / or the speed / velocity or vigor of the "shake" can also be components of the device's motion type. A motion type can include a single motion type or a combination of motion types, such as translational reciprocating motion and rotational motion, and the motion can be in multiple directions. A motion threshold can be set so that a motion type is identified from the motion data only when the speed or amplitude of the motion exceeds the threshold. This avoids incorrect motion type identification and, therefore, incorrect control operation execution. The motions can include a complex mix or collection of motions or motion types, such as a vibration and simultaneous rotation, or a sequence of motions, such as a vibration followed by a rotation, or a rotation in one direction followed by a rotation in another direction. Thus, the sequence of motions can be of the same motion type or a sequence of different motion types.

[0023] To reduce errors in the device's recognition of action types, during the learning phase, after receiving user input identifying an operation to be performed on information displayed in one or more identified display areas, the device may request that the user perform the action type more than once, so that the action can be detected more than once and the action pattern can be recorded each time, so that some form of average of the desired action pattern can be stored as data identifying the action type in the action type control data. This smoothing or averaging of the recorded actions that the user desires to be the action type for the action to be performed on information displayed in one or more identified display areas assists the processor in learning deviations in action data that may be received during the operation phase in which the user intends to cause an operation on information displayed in one or more identified display areas. Errors in matching the action data from the action detection configuration with the action data stored in the action type control data are reduced.

[0024] In the learning phase, a user may input a series of user inputs representing a series of desired action types to be used to perform corresponding operations. Thus, following the learning phase, user inputs may be received, action data may be input in pairs as a series to generate action type control data for a plurality of control operations to be performed on information displayed in one or more identified display areas.

[0025] The learning phase may receive a user selection of one or more display areas to identify the one or more display areas as part of the user input. The user selection of one or more display areas may be made using a pointer or touch input, or simply by selecting a display area from a list of display areas enumerated by identifier.

[0026] In the learning phase, menus may be displayed to the user to allow the user to generate user input by simply selecting one or more display areas to which operations should be applied and the menu items that are displayed.

[0027] In an alternative example, during the learning phase, the device may have a set list of control operations, and user input includes the option to implement the list to learn the action types for all enumerated operations to be performed on information displayed in one or more identified display areas. Thus, during the learning phase, the device simply requests that the user trigger the required action types for the device in sequence for the enumerated operations to be performed on information displayed in one or more identified display areas. The device may indicate to the user when each type of action should be performed for each control operation. This may be via a displayed message, an indicator light, or an audio output.

[0028] Action types and control operations can be learned for individual users. For example, a user can enter their name or sign in to use the display device, and then when the user enters their name or signs in, the learned action types and control operations are stored for that user for use.

[0029] A user may use each region of the display simultaneously. Furthermore, the display screen regions may be positioned as desired by the user, set by the user, or pre-set. A user may use motion controls to manipulate and reposition the display regions, for example, points to select and position or reposition portions of the display screen region or data within a selected display region.

[0030] In one embodiment, a user can select and control the display area, e.g., by tapping on a selected display area, by zooming in on a display area, or by expanding the display area as needed or desired by the user. This may allow a user to fully expand the display area, e.g., to cover the entire screen.

[0031] In one embodiment, the device includes a flexible device with one or more displays with one or more display areas, where display data can be shifted between one or more display areas, appearing on one or more sides of the device display, and shifting on one or more sides of the device display. A device with a display on both sides may include an electronic document reader that can display and control the displayed data. For example, data can be displayed on the front side A of the device and then shifted to the opposite side B of the device.

[0032] In one embodiment, the device includes a flexible device, where data may be displayed on one or more folded or curved regions of the flexible device, and the user may use motions or gestures to transition data between display regions on the same display or another display if more than one display is provided. User input configurations may include any conventional device input configuration, such as a keyboard, a pointing device (e.g., a mouse, trackpad, or tracker ball), a touch-sensitive display, and / or a pen. The pen allows the user to input handwritten text and drawings.

[0033] The device can have a display that can include a fixed or flexible display or multiple displays. The display or multiple displays can be removable. The multiple displays can be of different types, for example, one flexible and one rigid.

[0034] One aspect provides a carrier medium, such as a non-transitory storage medium storing code for execution by a processor of a machine to perform the method, or a transient medium carrying processor-executable code for execution by a processor of a machine to perform the method. An embodiment can be implemented in programmable digital logic that embodies computer code. The code can be provided to programmable logic, such as a processor or microprocessor, on the carrier medium. One such embodiment of the carrier medium is a transient medium, i.e., a signal, such as an electrical signal, an electromagnetic signal, an acoustic signal, a magnetic signal, or an optical signal. Another form of carrier medium is a non-transitory storage medium that stores code, such as a solid-state memory, a magnetic medium (hard disk drive), or an optical medium (compact disc (CD) or digital versatile disc (DVD)).

[0035] In one example, the device can be used as a display unit in any of the example display apparatuses disclosed in co-pending U.S. Patent No. 6,213,999, filed December 29, 2017, the entire text of which is incorporated herein by reference, or can be used with the device disclosed in co-pending U.S. Patent No. 6,213,999, filed on the same day as this application, entitled "Display Apparatus," the entire text of which is incorporated herein by reference.

[0036] Specific embodiments will now be described with reference to the drawings.

[0037] 1 is a schematic diagram illustrating a handheld device 1. Device 1 can include any handheld device, such as a mobile phone.

[0038] The motion directions that device 1 can undergo as a way of inputting commands for one or more independent display regions of the device are illustrated by arrows in Figure 1. The motion can be a linear or back-and-forth motion in any of three directions along three axes x, y, and z in three dimensions, as indicated by the crossed arrows. Additionally or alternatively, the motion can be a non-linear, curved, or rotational motion, as indicated by the curved arrows A and B. The rotational motion can be a rotation about either the x, y, or z axis.

[0039] FIG. 2 is a schematic diagram illustrating a mobile device 2. As shown in FIG.

[0040] The device 2 includes a display screen 3 capable of displaying information. In this example, the display screen 3 displays three types of information in three different display areas: typed text 11, handwritten text 41, and an image 31. While this embodiment illustrates different information being displayed in different areas, the invention also applies to displaying the same information type or a mixture of information types in the same or different display areas. The displayed information can be, for example, a document or form having areas to be completed or filled out, such as an address area and a name area in a standard form letter or fields in a form. The information displayed in the display areas can include any of text, images, or video. The information can be displayed by any application running on the device, such as a web browser, an email application, a word processor, a spreadsheet, an image editor, etc. Handwritten text 41 can be input using a pen device 50. The display screen 3 can be sensitive to the proximity of the pen device 50 to detect the coordinates of the pen device 50 and track its location, thus generating the input handwritten text 41. The display screen 3 may be touch sensitive, for example to allow input of information and commands into the device 2 by means of a touch sensitive optional display and / or keyboard.

[0041] In the example of FIG. 2, operation of device 2 may cause the execution of a control operation to modify any of the displayed information, for example, by erasing all of the displayed information or selective portions, such as one or more of typed text 11, handwritten text 41, or image 31, or by entering text or image data into the display area.

[0042] 3 illustrates an alternative display screen 20 of the device. In this example, there are three distinct display areas clearly defined by boundary markings: a typed text display area 10 displaying typed text 11, an image display area 30 displaying an image 31, and a handwritten text display area 40 displaying handwritten text 41 created using a pen 50.

[0043] 2 and 3 illustrate motion control of displayed information in the form of text or images, motion control can include any displayed information. In a shopping application, a virtual shopping cart can be moved, filled, or emptied, and items can be checked out. Each control operation can have an associated control action that the device is taught in a learning phase. Some control options can be pre-stored and do not require teaching.

[0044] The motion of the device in this example can be a back-and-forth linear or reciprocating motion in any of three directions along three axes x, y, and z in three dimensions, as indicated by the crossed arrows in Figure 1. Additionally or alternatively, the motion can be a non-linear, curved, or rotational motion, as indicated by the curved arrows in Figure 1.

[0045] 3, an action of the device may erase all or part of the information displayed in one or more of display areas 10, 30, and 40. One or more of the displayed areas and the operations performed on the information in the displayed areas are defined as commands specific to the device's action. The device stores a set of action patterns along with corresponding display area identifiers and information operations such that a particular action type causes a particular action on information displayed in only one display area, only a subset of the display areas, or all of the display areas.

[0046] Although different types of information are illustrated and described as being displayed in different display regions in Figure 3, the different display regions may display the same information type or any combination of information types. Also, the display regions may display a mix of information types. The example in Figure 3 is for illustrative purposes only.

[0047] The examples of Figures 2 and 3 describe control operations that control the display of information in particular display areas according to recognized patterns or types of movement.

[0048] The motion type detected by the motion detection configuration of the device can detect any one of multiple types of motion patterns in any dimension or direction. Examples of different motion patterns are front-to-back shaking, side-to-side shaking, up-and-down shaking, left-to-right rotation (side-to-side tilt), up-and-down rotation (up-and-down tilt), and rotation. Each of these represents a lateral motion (shaking) or rotational motion along or about one of the three axes illustrated in FIG. 1. Based on the initiating motion, two types of each of these motion types can be detected. For example, front-to-back can be front-to-back or back-to-front, side-to-side can be left-to-right or right-to-left, rotation can be clockwise rotation or counterclockwise rotation, etc. Furthermore, the vigor with which the motion is performed (e.g., the speed or frequency of shaking or reciprocity) can be used as a distinguishing feature for different motion types or patterns. Thus, the same spatial displacement of the display device can correspond to different motion types depending on the speed, acceleration, and frequency with which the displacement is performed. Additionally, complex motion types can include complex motion patterns that include combinations of basic motion types, such as lateral and rotational motions, or combinations of opposing lateral and rotational motions.

[0049] The device's memory can store a set of operations to be implemented on one or more display regions identified by the processor for each type of motion detected by the motion detection arrangement and identified by the processor from signals from the motion detection arrangement. These operations can be predefined or downloaded from a data source, they can be user-defined, or they can be supplemented. User definition or supplementation can occur during a learning phase.

[0050] FIG. 4 is a schematic diagram illustrating the electronic components of an exemplary device.

[0051] The processor 100 is connected to static / non-volatile memory 102, which stores code used by the processor 100, such as operating system code, for storing data and information for display in a non-volatile manner. Volatile memory 103 is provided for storing application code for implementation by the processor 100 and data and information for display. The volatile memory can store code (e.g., code modules) including code for identifying motion patterns detected by the motion sensor 104 and code (e.g., code modules) for controlling the display screen 105 in response to performing operations on information displayed in separate display areas. The memory can also store identifiers for display areas for motion types that were pre-stored or learned during a learning phase and data mapping operations on the display areas.

[0052] A display screen 105 is connected to the processor 100 for display of information. The display screen 105 may include a touch-sensitive display to provide an input mechanism. A pen 50 (shown in FIG. 2) may be used in conjunction with the display screen 105 to allow for the entry of freehand text or drawings. Optionally, a camera 106 may be connected to the processor 100 to capture images. The camera may also be used as an input mechanism for inputting information and commands using gesture recognition performed by the processor on images captured by the camera 106.

[0053] A network connector 101 is connected to the processor 100 for communication with a communication or computer network. This communication link can pass data, information, and control between the device and other devices or computers over the network. The network connector 101 can be connected to the communication or computer network using a wired or wireless connection. For example, the wired connection can be an Ethernet connection to a LAN or telephone connection, so that the network connector 101 acts as a modem or router. The wireless connection can be WIFI, WiMAX, CDMA, GSM, GPRS, wireless local loop, or WAN. For low-range and low-power wireless communication, Bluetooth™ may be used.

[0054] A motion sensor 104 is connected to the processor 100 to detect device motion. The motion sensor 104 may include any conventional motion sensor, such as an accelerometer. The motion sensor 104 may detect device motion in any direction. That is, the motion sensor 104 is a multi-axis motion sensor that detects motion along any of three orthogonal axes in three dimensions.

[0055] In any of the above examples, the device may have display screens on both sides so that the device can be flipped over to view information on both sides. Control operations can be applied to both sides or only to one side. In one example, this can be selectively configured. The display or displays can also be detached from the mounting frame.

[0056] In one or more embodiments, the device can include a docking portion or mount to which the display screen is attached in a detachable or demountable manner so that the display screen can be used separately from the mount. The mount and display screen can communicate between each other using a wireless link, so that the mount can include some components of a display device, such as a processor and storage, while the display screen can include only the components necessary to operate the display screen, detect display screen motion, and communicate with the processor. In one embodiment, more than one display screen can be provided detachably from the mount, with each display screen operating independently to provide motion-detected erase operations using a shared processor in the mount.

[0057] The device may include one or more loudspeakers for audio output and one or more microphones for audio input, and may be connected to peripheral devices such as a keyboard and / or pointing device by wired or wireless connections to allow input of information and commands into the device.

[0058] A method of operating the exemplary device will now be described with reference to Figures 5 and 6.

[0059] FIG. 5 is a flow diagram illustrating the learning phase of a device.

[0060] In step S1, the device enters a learning phase. This can be automatic upon device startup or as a result of other device operation, or it can be automatic as a result of user input, such as user selection, i.e., using the device's input configuration. In step S2, the device's display displays a menu of defined operations to be performed on information displayed in one or more identified display areas, which the user can select to perform using the action as user input. The displayed menu can, for example, be a simple list or a drop-down menu. The list can be organized into categories of operation type. The menu can display a list of display areas to which the operation can be applied, and the user can select one or more display areas. Alternatively, the display can display the areas in a selectable manner so that the user can select one or more areas to which the selected operation should be applied, using, for example, a pointer device or touch input to select an area or areas of the display.

[0061] In step S3, a user selection of an operation menu item is received. Then, in step S4, the device waits for a motion detector, e.g., an accelerometer, to detect device motion. The motion is recorded over a short period of time, e.g., 1-2 seconds, suitable for capturing a motion pattern. To improve the accuracy of the matching process and allow for variations in the user's motion pattern, in step S5 the process loops back and requests the user to repeat the motion input several (N) times, e.g., anywhere from 3-10 times. More repetitions improve the accuracy of the matching process but reduce the usability of the device; i.e., the user becomes impatient with the requirement to repeat too many motion patterns.

[0062] At the end of the loopback process, in step S6, the recorded motion patterns for the repeated recordings are averaged, and the average is used for storage in step S7 as part of the information about the motion type in the stored motion type control data. Data indicative of the degree of deviation between the recorded motion patterns can also be stored as part of the motion type data to assist the matching process during the operation phase, i.e., to assist in assessing whether a match is within the expected deviation of the stored average motion pattern. An identifier identifying the corresponding control operation is also stored as part of the stored motion type control data.

[0063] The process then returns to step S2 to allow the user to select another operation from the menu. Once the user has finished selecting operations, the user can choose to end the learning phase.

[0064] FIG. 6 is a flow diagram illustrating the device operation phase that follows the learning phase.

[0065] In step S10, the device enters the operation phase, and in step S11, a device operation is detected. In step S12, the pattern of the operation is compared with a pattern stored for the operation type in the operation type control data to determine whether the operation matches the operation type. If a match is not found (step S13), the process returns to step S11 to detect a further operation. If an operation is found to match the operation data for the matching type in step S13, a corresponding operation on the information displayed in one or more identified display areas is identified in the operation type control data in step S14, and the operation is performed on the information displayed in one or more identified display areas in step S15. The process then returns to step S11 to detect a next operation of the device for a further operation on the information displayed in one or more identified display areas.

[0066] In Figure 5, the motion data for multiple recorded motion maneuvers is averaged. However, the invention is not limited to averaging, and only one motion detection may be used, or multiple may be recorded and stored individually for individual matching with motion patterns during a maneuver phase.

[0067] In Figure 5, the user is presented with a menu to select an operation that will tell the device the type of action for that control operation. Using user input configuration, users can define their own operations to be performed on information displayed in one or more identified display areas.

[0068] In any of the above examples, the device may include other input configurations, such as an audio sensor for audio input, a keyboard, a pointer device (e.g., a mouse), or a touch screen, to allow a user to input control and information in connection with a motion-sensing control input method. A camera may also be provided for image capture to enable gesture input as a method of control or information input. The device may include an infrared sensor, such as a passive infrared sensor (PIR).

[0069] The information displayed in the display area can include video (movie, television, 3D video, panoramic view, etc.).

[0070] The device may include a remote control for controlling the device, the remote control being operable in response to operational control of the device.

[0071] In embodiments, in addition to or instead of motion control input, a camera can be provided in the device to capture video in a direction facing the user so that the user can use gestures to input controls depending on the identified gesture type. Gesture control can include recognition of movements of the user's body parts, i.e., hands, facial features, or overall body position. Pattern recognition techniques can be used to recognize and identify gesture types. Exemplary gesture types can include sweeping the hand left or right, sweeping up or down, or waving the hand.

[0072] If gesture control is provided along with motion control, a gesture control clearing operation can be used, for example, when a user is unable to pick up the device, for example, when the user has wet or dirty hands. Furthermore, the device can be controlled to respond to a detected gesture type by identifying an equivalent or related motion type that inputs the same control and by controlling the display screen to mimic an equivalent motion to the gesture command. For example, if the motion command is a left-to-right rotation, when an equivalent or related gesture command, such as waving a hand from left to right, is identified, the display screen can rotate the display from left to right to act as confirmation to the user that the gesture command was recognized as a motion command. Equivalent gesture types can be automatically determined by a processor from the corresponding motion types, and thus the motion type data can include corresponding gesture type data for the same operation. Using gestures that match motion commands in common movements means that the gestures are intuitive to users. For example, rotating a device can be considered equivalent to rotating a user's hand.

[0073] This aspect of the invention can be applied to inputting any type of command to a device, including running an application, taking a picture using the device's camera, or any other operation that changes the state of the device. This aspect can use the hardware of Figure 4 and the methods of the flow diagrams of Figures 5 and 6 to learn and execute control operations. This aspect is not limited to controlling display operations in a display area, but instead can be applied to inputting any control operation to a device.

[0074] Action types and control operations can be learned for individual users. For example, a user can enter their name or sign in to use the display device, and the learned action types and operations are then stored for that user for use when the user enters their name or signs in.

[0075] In one aspect, the present invention provides gesture control to complement motion control. In this aspect, a device includes a memory for storing control data including an identification of a motion type, a corresponding control operation, and corresponding gesture type data, a motion detection arrangement for detecting motions of the device, a gesture detection arrangement for detecting gestures, a user input arrangement for receiving user input identifying a control operation during a learning phase, and a processor programmed to: identify a motion type from motions detected by the motion detection arrangement when receiving the user input identifying the control operation during the learning phase, store the received user input corresponding to the data identifying the motion type in the control data, and determine corresponding gesture type data; and, following the learning phase, use the control data to identify the motion type and perform the corresponding control operation when the motion detection arrangement detects a motion, or use the control data to identify the gesture type and perform the corresponding control operation when the gesture detection arrangement detects a gesture.

[0076] In this aspect, the device can be mobile or fixed / mounted. A camera can be provided in the device to capture video in a direction facing the user so that the user can use the gesture as a control input depending on the type of gesture identified. Gesture control can include recognition of the movement of the user's body parts, i.e., hands, facial features, or general body position, or recognition of an object moved by the user, such as a stick, ruler, or something worn by the user. Pattern recognition techniques can be used to recognize and identify gesture types. Exemplary gesture types can include sweeping the hand left or right, sweeping up or down, or waving the hand. The gestures match the device's movements learned in the learning phase. For example, rotating the device can be mimicked by rotating the user's hand.

[0077] It will be readily apparent to those skilled in the art that various other changes in the details, materials, and arrangements of the components and method steps described and illustrated to explain the nature of the inventive subject matter may be made without departing from the principles and scope of the inventive subject matter as set forth in the appended claims. [Prior art documents] [Patent documents]

[0078] [Patent Document 1] UK Patent Application No. 1722249.8 [Patent Document 2] UK Patent Application No. 1805278.7

Claims

1. A device, a memory for storing control data including an identification of an action type, a corresponding control operation, and corresponding gesture type data; a motion detection configuration for detecting motion of the device; a gesture detection arrangement for detecting a gesture; a user input arrangement for receiving user input identifying a control operation during a learning phase; a processor, the processor comprising: During the learning phase, when a user input identifying a control operation is received, identifying an action type from the action detected by the action detection arrangement, storing the received user input corresponding to data identifying the action type in the control data, and determining corresponding gesture type data; following the learning phase, when the action detection arrangement detects an action, it uses the control data to identify an action type and execute the corresponding control operation, or when the gesture detection arrangement detects a gesture, it uses the control data to identify a gesture type and execute the corresponding control operation. device.

2. The device of claim 1 , wherein the gesture detection arrangement includes a camera.

Images