KEYPAD INPUT DEVICE, KEY GESTURE EVALUATION DEVICE, KEY GESTURE EVALUATION METHOD AND KEY GESTURE EVALUATION PROGRAM

Abstract

Keypad input device (1, 2), comprising: a keypad (20) that displays an operational image on a screen, receives a keystroke operation performed by a user, and outputs operational information corresponding to the keystroke operation; and a tactile gesture evaluation device (10, 10a) that receives the operation information and generates a choice value based on the operation information, wherein the tactile gesture assessment device (10, 10a) contains: a line-of-sight presence / absence assessment unit (12) that receives input information according to the touch gesture operation, assesses on the basis of the input information whether the touch gesture operation is an operation in a line-of-sight presence state in which the user's line of sight is directed towards the screen, or an operation in a line-of-sight absence state in which the user's line of sight is not directed towards the screen, and outputs line-of-sight assessment information indicating a result of the assessment; an operation mode switching unit (13) that outputs command information based on the line-of-sight assessment information; an operations evaluation unit (14) that generates the selection value based on the operations information through an evaluation procedure according to the command information; and a screen control unit (16) that causes the keypad to display an image according to the command information as the operation image, and the operational evaluation unit (14) contains: a screen component gesture evaluation unit (141) that determines the selection value based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touch panel in the line-of-sight presence state; and a full-screen gesture evaluation unit (142) that determines the selection value based on the operation information according to the touch gesture operation performed on the entire screen of the touch field in the sight-line absence state.

Description

INVENTION AREA

[0001] The present invention relates to a keypad input device that receives a keystroke operation and outputs a signal according to the keystroke operation, and relates to a keystroke evaluation device, a keystroke evaluation method and a keystroke evaluation program for receiving operation information according to the keystroke operation as an input and outputting a signal according to the received operation information. GENERAL STATE OF THE ART

[0002] Generally, a user employing a keypad input device performs a keystroke operation while viewing a GUI (Graphical User Interface) screen displayed on the keypad. While performing a task such as driving a motor vehicle or adjusting the position of a camera, the user cannot perform the keystroke operation while viewing the keypad screen, that is, in a "line-of-sight presence state," meaning a state in which the user's eyes are focused on the keypad screen.

[0003] As a countermeasure to this problem, patent reference 1 proposes a device that determines an operational part (hereinafter referred to as a "screen component") to be displayed on a touchpad based on the degree of similarity between a shape formed by connecting contact points of several fingers contacting the touchpad and a previously set shape.

[0004] Furthermore, patent reference 2 proposes a device that assesses whether the user is a person (e.g., a visually impaired person) performing a tactile gesture operation without looking at a touchpad screen, that is, in a “sight-line absence state” as a state in which the user does not direct their eyes to the touchpad screen, when the area of ​​a region specified by the positions of several fingers contacting the touchpad is greater than or equal to a previously set threshold. REFERENCE ACCORDING TO THE STATE OF THE TECHNOLOGY PATENT REFERENCE Patent reference 1: WO 2016 / 035207 A1 Patent reference 2: Japanese patent application Publication number JP 2016-020267 A BRIEF DESCRIPTION OF THE INVENTION AND THE PROBLEM TO BE SOLVED BY THE INVENTION

[0005] However, with the devices described in patent references 1 and 2, the user must focus his attention on a tactile gesture operation to form a specific shape with several fingertips, and thus there are cases where the level of attention decreases to another task performed in parallel with the tactile gesture operation.

[0006] With the device described in patent reference 1, the user still has to perform a touch gesture operation that uses the screen component displayed on the keypad without looking at the screen of the keypad, and thus there are cases where it is difficult to perform the correct touch gesture operation on the screen component.

[0007] Furthermore, with the setup described in patent reference 2, the user must perform the touch gesture operation in an operationally effective area of ​​the keypad, and thus there are cases where the user cannot enter corresponding operation information while not looking at the keypad screen.

[0008] An object of the present invention is to provide a touchpad input device on which an input operation can be easily and correctly performed by means of a touch gesture operation by means of a corresponding switching between an operation mode in a line-of-sight presence state, in which the touch gesture operation is performed using a screen component displayed on the touchpad, and an operation mode in a line-of-sight absence state, in which the touch gesture operation is performed by using the entire screen of the touchpad as a touch gesture operation receiving area, and to provide a touch gesture evaluation device, a touch gesture evaluation method and a touch gesture evaluation program to enable it toto easily and correctly perform an input operation by means of a keystroke operation using a corresponding switch between the operation mode in the line-of-sight presence state and the operation mode in the line-of-sight absence state. MEANS TO SOLVE THE PROBLEM

[0009] A keypad input device according to one aspect of the present invention is a device comprising a keypad that displays an operation image on a screen, receives a keystroke operation performed by a user and outputs operation information corresponding to the keystroke operation, and a keystroke evaluation device that receives the operation information and generates a selection value based on the operation information.The touch gesture evaluation unit includes a line-of-sight presence / absence evaluation unit that assesses whether the touch gesture operation is an operation in a line-of-sight presence state, where the user's line of sight is directed at the screen, or an operation in a line-of-sight absence state, where the user's line of sight is not directed at the screen, and outputs line-of-sight evaluation information indicating the result of the assessment; an operation mode switching unit that outputs command information based on the line-of-sight evaluation information; an operation evaluation unit that generates the choice value based on the operation information through an evaluation procedure according to the command information; and a screen control unit that causes the touch panel to display an image according to the command information as the operation image.The operation evaluation unit includes a screen component gesture determination unit, which determines the selection value based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touchpad in the line-of-sight presence state, and a full-screen gesture determination unit, which determines the selection value based on the operation information according to the touch gesture operation performed on the entire screen of the touchpad in the line-of-sight absence state.

[0010] A touch gesture evaluation method according to another aspect of the present invention is a method of receiving operation information from a touch field that displays an operation image on a screen, receiving a touch gesture operation performed by a user, and outputting the operation information according to the touch gesture operation, and generating a selection value based on the operation information.The procedure includes: a line-of-sight presence / absence assessment step of assessing whether the touch gesture operation is an operation in a line-of-sight presence state, where the user's line of sight is directed at the screen, or an operation in a line-of-sight absence state, where the user's line of sight is not directed at the screen, and outputting line-of-sight assessment information indicating a result of the assessment; an operation assessment step of generating the choice value based on the operation information by means of an assessment procedure according to command information based on the line-of-sight assessment information; and a screen control step of causing the touchpad to display an image according to the command information as the operation image.In the operation evaluation step, the choice value is determined based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touchpad in the line-of-sight presence state, and the choice value is determined based on the operation information according to the touch gesture operation performed on the entire screen of the touchpad in the line-of-sight absence state. EFFECT OF INVENTION

[0011] According to the present invention, by appropriately switching between the operation mode in the line-of-sight presence state, in which the touch gesture operation is performed using a screen component displayed on the touchpad, and the operation mode in the line-of-sight absence state, in which the touch gesture operation is performed by using the entire screen of the touchpad as the touch gesture operation reception area, an input operation can be easily and correctly performed by a touch gesture operation. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a functional block diagram showing a schematic representation of a keypad input device according to a first embodiment of the present invention. Fig. Figure 2 is a diagram showing an example of a hardware implementation of the keypad input device according to the first embodiment. Fig.3(a) and Fig. 3(b) are diagrams showing examples of a screen of a keypad in the keypad input device according to the first embodiment. Fig. Figure 4 is a flowchart showing an operation of a touch gesture evaluation device (touch gesture evaluation procedure) in the touchpad input device according to the first embodiment. Fig. Figure 5 is a diagram showing an example of a screen of a keypad in a keypad input device according to a modification of the first embodiment. Fig. Figure 6 is a functional block diagram showing a schematic representation of a keypad input device according to a second embodiment of the present invention. Fig. Figure 7 is a flowchart showing an operation of a touch gesture evaluation device (touch gesture evaluation procedure) in the touchpad input device according to the second embodiment. Fig. Figure 8 is a diagram showing a method for obtaining history information about touch gesture operations in the touchpad input device according to the second embodiment. Fig. Figure 9 is a diagram showing an example where an intended input cannot be made if the touch gesture operation is performed in a line-of-sight absence state, which is a state in which the touch panel screen is not viewed in the touch panel input device. Fig. Figure 10 is a diagram showing an example of a touch gesture operation performed in the line-of-sight absence state, a state in which the screen of the touch panel in the touch panel input device according to the second embodiment is not viewed. Fig.Figure 11 is a diagram showing an example in which the direction, angle and image scaling extent of a camera in a camera image acquisition device communicating with the keypad input device according to the second embodiment are adjusted without looking at the screen of the keypad input device. Fig. Figure 12 is a functional block diagram showing a schematic representation of a keypad input device according to a third embodiment of the present invention. Fig. Figure 13 is a diagram showing an example of a hardware implementation of the keypad input device according to the third embodiment. Fig. Figure 14 is a flowchart showing an operation of a touch gesture evaluation device (touch gesture evaluation procedure) in the touchpad input device according to the third embodiment. Fig. 15(a) and Fig.15(b) are diagrams showing a method for assessing a line-of-sight presence state and a line-of-sight absence state in the keypad input device according to the third embodiment. MODE FOR EXECUTING THE INVENTION

[0012] Embodiments of the present invention are described below with reference to the accompanying drawings. In the following description, a keypad input device comprises a keypad with a screen used for touch-based operations (operation screen) and a touch gesture evaluation device that receives operation information on the keypad.The keypad input device is installed in an object device or communicatively connected to the target device, making it applicable to an operating screen of an electrical device as the object device, an operating screen of a camera as the object device, an operating screen of a factory device as the object device, an operating screen installed in a motor vehicle, a marine vessel, an aircraft, or the like as the object device, an operating screen of a mobile information terminal such as a smartphone or tablet terminal as the object device, and so on. The keypad input device is capable of receiving a signal (e.g.,to deliver a selection value) based on operational information entered on the operating screen of the keypad by means of a keystroke operation (also referred to as a "keystroke operation") to the object device equipped with the keypad input device or the object device that can communicate with the keypad input device.

[0013] The touchpad is a tactile gesture input device that receives the tactile gesture operation performed by the user. A tactile gesture operation is an information input operation performed by a specific movement of the user's fingers (or palm, or fingers and palm) or similar action. The tactile gesture operation can include tapping, which involves lightly striking the touchpad's operating screen with a finger; swiping, which involves stroking the touchpad's operating screen with a finger; and wiping, which involves sliding a finger across the touchpad's operating screen.Furthermore, the touch gesture operation can include dragging (the operation of dragging a screen component on the touchpad with a finger), pinching (the operation of decreasing the distance between several fingers while touching the touchpad screen), pinching apart (the operation of increasing the distance between several fingers on the touchpad screen), and so on. Incidentally, the touch gesture operation can also include operation using a stylus as a pen-like input aid. (1) First embodiment(1-1) design

[0014] Fig. Figure 1 is a functional block diagram showing a schematic representation of a keypad input device 1 according to a first embodiment of the present invention. As shown in Fig.As shown in Figure 1, the keypad input device 1 according to the first embodiment includes a touch gesture evaluation device 10 and a keypad 20. The touch gesture evaluation device 10 is a device capable of executing a touch gesture evaluation procedure and a touch gesture evaluation program according to the first embodiment.

[0015] As in Fig. As shown in Figure 1, the keypad 20 contains an operation field unit 21, which receives a keystroke operation performed by the user and outputs operation information (also referred to as "keystroke information") A0 according to the keystroke operation, and a screen field unit 22, which is arranged in superposition with the operation field unit 21 and is capable of displaying an operation image such as a GUI screen. The screen field unit 22 is, for example, a liquid crystal display.

[0016] As in Fig.As shown in Figure 1, the tactile gesture evaluation unit 10 includes an operation information input unit 11, a line-of-sight presence / absence evaluation unit 12 as an operation mode evaluation unit, an operation mode switching unit 13, an operation evaluation unit 14, a notification unit 15, and a screen control unit 16. The operation evaluation unit 14 includes a screen component gesture evaluation unit 141 and a full-screen gesture evaluation unit 142.

[0017] The operation information input unit 11 receives the operation information (operation signal) A0, which is output by the operation field unit 21. The operation information input unit 11 outputs input information A1, corresponding to the received operation information A0, to the line-of-sight presence / absence assessment unit 12 and the operation assessment unit 14. The input information A1 is information corresponding to the operation information A0 and can also be the same information as the operation information A0.

[0018] The line-of-sight presence / absence assessment unit 12 assesses, based on the input information A1 received from the operation information input unit 11, whether the touch gesture operation is an operation performed by the user while viewing the screen of the keypad 20, i.e., an operation in a line-of-sight presence state as a state in which the user's line of sight is directed at the screen of the keypad 20, or the touch gesture operation is an operation performed by the user while not viewing the screen of the keypad 20, i.e., an operation in a line-of-sight absence state as a state in which the user's line of sight is not directed at the screen of the keypad 20.An operation mode assuming a line-of-sight presence state is, for example, an operation mode for performing a touch gesture operation on a screen component displayed on the touchpad 20. An operation mode assuming a line-of-sight absence state is, for example, an operation mode in which the entire screen of the touchpad 20 is used as an operationally effective area for receiving the touch gesture operation.

[0019] The line-of-sight presence / absence assessment unit 12 sends the line-of-sight assessment information A2, which indicates the result of the assessment of whether the touch gesture operation is an operation in the line-of-sight presence state or an operation in the line-of-sight absence state, based on the input information A1 according to the touch gesture operation by the user, to the operation mode switching unit 13. This assessment can be performed, for example, by the line-of-sight presence / absence assessment unit 12 including a storage unit (e.g., a memory in Fig. 2, which will be described later), which stores previously determined patterns of touch gesture operations, based on the degree of similarity between the touch gesture operation specified by the input information A1 and the patterns of touch gesture operations stored in the storage unit.

[0020] The operation mode switching unit 13 outputs command information A3 for switching (setting) the screen content on the screen of the keypad 20, the procedure for evaluating the input information A1 according to the key gesture operation on the keypad 20 by the operation evaluation unit 14, and the procedure for notification by the notification unit 15 based on the line-of-sight presence / absence evaluation information A2 received by the line-of-sight presence / absence evaluation unit 12.

[0021] The operation evaluation unit 14 receives the command information A3 from the operation mode switching unit 13 and switches the procedure for evaluating the input information A1 according to the command information A3.

[0022] The full-screen gesture evaluation unit 142 evaluates the touch gesture operation across the entire screen of the touchpad 20 and determines a selection value A7 as an output signal based on the input information A1. The evaluation of the touch gesture operation across the entire screen of the touchpad 20 is used in a limited small area on the touchpad 20 (as part of the screen of the touchpad 20) in a touch gesture operation in the line-of-sight presence state, and in a large area on the touchpad 20 (the entire screen of the touchpad 20) in a touch gesture operation in the line-of-sight absence state.

[0023] The screen component gesture evaluation unit 141 evaluates the touch gesture operation based on a screen component as an operation image displayed by the screen field unit 22 and the input information A1, and determines the selection value A7 as an output signal based on the screen component and the input information A1. The evaluation of the touch gesture operation on the screen component on the touch field 20 is used in a large area on the touch field 20 (the entire screen of the touch field 20) in a touch gesture operation in the line-of-sight presence state, and in a limited small area (a part of the screen of the touch field 20) in a touch gesture operation in the line-of-sight absence state.

[0024] The screen control unit 16 outputs an image signal A6 indicating the operating image to be displayed on the screen field unit 22 of the keypad 20. The screen control unit 16 changes the screen content of the screen field unit 22 according to the operating evaluation information A4 received from the operating evaluation unit 14 and the command information A3 received from the operating mode switching unit 13.

[0025] Notification Unit 15 switches the method of notifying the user of information depending on whether the touch gesture operation is a touch gesture operation in the line-of-sight presence state or a touch gesture operation in the line-of-sight absence state. Notification Unit 15 issues a notification or emits a notification signal containing notification content according to command information A3. For example, Notification Unit 15 notifies the user of the status of the touch gesture operation using sound, a screen display, vibration of a vibrator, illumination of a lamp, or the like, according to input information A1 received from Operation Evaluation Unit 14.As above, the notification unit 15 modifies the notification content according to the operation assessment information A4 received from the operation assessment unit 14 and the command information A3 received from the operation mode switching unit 13. If the notification by the notification unit 15 is an audible notification, the notification unit 15 outputs a notification signal to a loudspeaker as an audio output unit. The loudspeaker is located in . Fig. Figure 2 shows what will be explained later. If the notification from the notification unit 15 is an image display, the notification unit 15 sends notification information A5 to the screen control unit 16, and the screen control unit 16 transmits an image signal based on the notification information to the screen field unit 22.

[0026] The selection value A7 issued by the operation evaluation unit 14 is a selection value determined by the operation evaluation unit 14 on the basis of the input information A1, and an application program or the like of the device equipped with the keypad input device 1 performs a device control or the like on the basis of the selection value A7.

[0027] Fig. Figure 2 is a diagram showing an example of a hardware (H / W) implementation of the keypad input device 1 according to the first embodiment. As shown in Fig. As shown in Figure 2, the keypad input device 1 according to the first embodiment comprises the keypad 20, a processor 31, a memory 32, and a loudspeaker 33. The in Fig.The touch gesture evaluation device 10 shown in Figure 1 can be implemented using memory 32 as a storage device for storing the touch gesture evaluation program as software and processor 31 as an information processing unit for executing the touch gesture evaluation program stored in memory 32 (e.g., by a computer). In this case, components 11 to 16 correspond to the Fig. 1 the processor 31, which runs the touch gesture evaluation program in Fig. 2 executes. Incidentally, it is also possible to execute part of the in Fig. 1 shown tactile gesture assessment device 10 by the in Fig. to implement the memory 32 shown in 2 and the processor 31, which executes the keystroke evaluation program.

[0028] The loudspeaker 33 is an audio output unit used to notify the user of the status of a keystroke operation performed in the line-of-sight absence state, using sound such as a voice announcement. The keypad input device 1 may include an additional device, such as a vibrator, a lamp, or a transmission device for wirelessly transmitting a notification signal, either instead of or as an additional component.

[0029] Fig. 3(a) and Fig. 3(b) are diagrams showing examples of the touch gesture operation and the screen of the touch panel 20 in the touch panel input device 1 according to the first embodiment.

[0030] Fig. Figure 3(a) shows an example of the screen of the touch panel 20 in an operating mode in the line-of-sight presence state. Fig.Figure 3(a) shows an example of the screen in a case where the line-of-sight presence / absence assessment unit 12 judges that the operating mode is the line-of-sight presence state based on the input information A1 specifying the touch gesture operation. The user can toggle the screen content by performing a swipe operation of the screen left or right with a finger. List 231 is a list that is displayed when the line-of-sight presence / absence assessment unit 12 judges that the state is the line-of-sight presence state. List 231 consists of several buttons (numbered rectangular areas), and List 231 can be moved up or down by swiping (sliding) a finger across List 231.For example, touching a button 232 with a finger can cause the line-of-sight presence / absence assessment unit 12 to assess that the state is the line-of-sight presence state, and the selection value is determined (finalized) by tapping a button (touching a button with a finger and lifting the stationary finger from the button within a predetermined time).

[0031] Fig. Figure 3(b) shows an example of the screen of the touch panel 20 in an operating mode in the line-of-sight absence state. Fig.Figure 3(b) shows an example of the screen in a case where the line-of-sight presence / absence assessment unit 12 assesses that the operation mode is the operation mode in the line-of-sight absence state based on the input information A1 specifying the touch gesture operation. For example, a delete area 241 in the line-of-sight absence state is a rectangular area surrounding ">>Delete>>", and when a finger is moved (slid) in the rectangular area in a predetermined direction, namely from left to right, then the operation mode in the line-of-sight absence state is deleted and switched to the operation mode in the line-of-sight presence state. (1-2) Operation

[0032] Fig.Figure 4 is a flowchart showing the operations of the touch gesture evaluation device 10 (touch gesture evaluation procedure) in the touchpad input device 1 according to the first embodiment.

[0033] In step ST101, the operation mode switching unit 13 sets an initial operation mode to the line-of-sight mode. At this time, the operation mode switching unit 13 outputs the command information A3, causing the screen of the keypad 20 to switch to a screen for the line-of-sight mode, and the input method of the keypad 20 to an input method using a screen component that uses the line-of-sight mode. It is also possible to set the initial operation mode to the line-of-sight mode.

[0034] In the next step ST102, the operation information input unit 11 receives the operation information A0 (coordinates indicating the position of a finger contact point, a contact condition of a finger, finger identification numbers relating to multiple contact points, etc.) from the keypad 20 and stores the operation information A0 in a storage unit (e.g., the one in Fig. 2 shown memory 32).

[0035] In the next step ST103, the line-of-sight presence / absence assessment unit 12 performs an assessment as to whether the tactile gesture operation is an operation in the line-of-sight presence state or an operation in the line-of-sight absence state, that is, a line-of-sight presence / absence assessment (assessment of the mode of operation) based on history information about the operation information A0 stored in the storage unit (information regarding the finger contact point, etc.).

[0036] In step ST103, the line-of-sight presence / absence assessment unit 12 assesses that the state is the line-of-sight absence state if any of the following assessment conditions (1) to (3) are met, for example, and assesses that the state is the line-of-sight presence state otherwise. Assessment condition (1): A finger is in contact with a point on the screen of the touch panel 20, and the contact position does not move for a time greater than or equal to a predetermined threshold. Assessment condition (2): Several fingers are in contact with several contact points on the screen of the touch panel 20 and remain in contact for a time greater than or equal to a predetermined threshold. Assessment condition (3): A palm of the hand remains in contact with the screen of the touch field 20 (namely, a large area greater than or equal to a predetermined threshold remains in contact for a time greater than or equal to a predetermined threshold). However, it is possible to use a different criterion for assessment than the criterion for assessment.

[0037] If the assessment condition (3) for the line-of-sight absence state is met in step ST103, there is a possibility that the coordinates of multiple detected contact points and the number of contact points are unstable (i.e., vary significantly over time). Therefore, it is also possible to use the history information on the operation information A0 (input information A1) regarding a period from the present time to a time a predetermined period earlier and to assess that the contact is a palm contact based on the degree of displacement at the coordinates (movement distance of the coordinates) of each contact point in that period (if, for example, the greatest movement distance exceeds a predetermined threshold).

[0038] If an operation of placing a palm in contact with the screen is performed in the assessment condition (3) for the line-of-sight absence state in step ST103, the operation information A0 from touchpad 20 contains information indicating that the contact condition is abnormal. Therefore, it is also possible to use the history information on the operation information A0 for a period from the present time to a time prior to a predetermined period and to assess that the contact is a palm contact if there is a history record indicating an abnormal contact condition during that period.

[0039] If the operation of placing the palm in contact with the screen in the evaluation condition (3) for the line-of-sight absence state in step ST103 continues to be performed, there is a possibility of a misjudgment that a finger is disconnected from the screen of the touchpad 20, and consequently a misjudgment that the touch gesture operation is a tap on a button as a screen component on the screen of the touchpad 20. Thus, it is also possible to use the history information on the operation information A0 regarding a period from the present time to a time before a predetermined period, to insert a standby process of checking whether an unstable tap (i.e.,a situation in which the number and positions of contact points change significantly over time) is not detected or no notification of an abnormal contact condition is received, for a predetermined time since all fingers separated from the touchpad 20 during that period, and to assess that the contact is a palm contact when such a condition is detected.

[0040] In step ST103, the line-of-sight presence / absence assessment unit 12 judges that the line-of-sight presence state is present when a finger makes contact with the screen and moves exactly as shown on the screen. For example, with respect to the screen in the Fig.3(b) the line-of-sight absence state shown, the line-of-sight presence / absence assessment unit 12, that the operation on the screen is a toggle operation to the line-of-sight presence state when the operation is performed to cause a finger to start typing within the deletion area 241 in the line-of-sight absence state, slide from the left mark ">>" to cross the right mark ">>" over the "Delete" characters and then detach from the screen, or to cause a finger to slide from the left mark ">>" to cross the right mark ">>" over the "Delete" characters and then cross the right boundary line of the deletion area 241.Incidentally, when assessing the state via the switching operation to the line-of-sight presence state in step ST103, it is also possible not to assess that the state is the line-of-sight presence state if a finger constantly moves from outside the extinguishing area 241 to inside the extinguishing area 241 in the line-of-sight absence state, a finger moves upwards or downwards out of the extinguishing area 241 while moving within the extinguishing area 241 in the line-of-sight absence state, or a finger moves to the right within the extinguishing area 241 and then moves back to the left in the line-of-sight absence state.

[0041] Furthermore, the condition for line-of-sight presence / absence assessment unit 12 to assess that the operation is the toggle operation to the line-of-sight presence state in step ST103 is not limited to the case where a finger is from the left mark ">>" in Fig.3(b) slides to cross the right mark ">>" over the marks "Delete", but may also include a case where a finger touches the deletion area 241 and moves to the right for a predetermined distance or more in the sight-out state.

[0042] If there is a change in the line-of-sight presence / absence assessment in step ST104 (YES), the process proceeds to steps ST105 to ST108, and the operation mode switching unit 13 issues command information A3 to switch the input method based on the switching of the operation screen on the keypad 20 and to switch the notification method of the notification unit 15. If there is no change in the line-of-sight presence / absence assessment in step ST104 (NO), the process proceeds to step ST109.

[0043] In step ST105, the operation evaluation unit 14 receives the command information A3 from the operation mode switching unit 13 and issues a command to switch to a valid input method for the whole screen, and the full screen gesture evaluation unit 142 sets the valid input method for the whole screen.

[0044] In the next step, ST106, the operation evaluation unit 14 receives the command information A3 from the operation mode switching unit 13 and issues a command to switch to the valid input method for the screen component, and the screen component gesture evaluation unit 141 sets the valid input method for the screen component. The processing order of steps ST105 and ST106 can be reversed.

[0045] In the next step, ST107, the screen control unit 16 receives the command information A3 to switch the screen from the operation mode switching unit 13 and switches the screen. For example, if the screen control unit 16 receives the command information A3 to switch the screen when the line-of-sight presence state is active, the screen control unit 16 displays the screen in the Fig. 3(a) line-of-sight presence state shown, as an example; if the state is the line-of-sight absence state, the screen control unit 16 displays the screen in which in Fig. 3(b) the line-of-sight absence state shown, as an example.

[0046] In the next step, ST108, the notification unit 15 receives the command information A3 to switch the notification content from the operation mode switching unit 13 and switches the notification content. For example, the notification unit 15 outputs a voice announcement such as "If you press and hold the screen, the operation mode switches to the mode for operating without looking at the screen" from the speaker in the case of the line-of-sight presence state. If the state switches to the line-of-sight absence state, a voice announcement is made such as "Due to the press-and-hold operation, the screen switches to an operation screen for a disabled person."When the state switches to the line-of-sight absence state, the speaker emits a voice announcement such as, “Touching the top of the screen increases the dial value; touching the bottom of the screen decreases the dial value. A two-point tap finalizes the dial value.” In this case, notification unit 15 can provide a voice announcement of the current dial value whenever a tap gesture is made on the top or bottom of the screen.

[0047] In step ST108, the announcement by the notification unit 15 is not limited to a voice announcement; it is also possible to make the announcement using a screen device other than the operated keypad 20, by making a voice announcement from a smartwatch, by vibrating the keypad, and so on.

[0048] In step ST109, the Operational Evaluation Unit 14 determines (finalizes) the choice value based on the input information. For example, if a displayed button is tapped in the line-of-sight presence state of Operational Mode, Operational Evaluation Unit 14 determines the number displayed on the button as the choice value. In contrast, in the line-of-sight absence state of Operational Mode, Operational Evaluation Unit 14 increments the choice value when a touch occurs on the top of the screen and decrements the choice value when a touch occurs on the bottom of the screen. Operational Evaluation Unit 14 finalizes the choice value when a two-point touch occurs and the fingers are simultaneously removed from the screen.When a touch gesture operation is received by the operation evaluation unit 14, the screen control unit 16 changes the screen content according to the touch gesture operation, and the notification unit 15 performs a notification about the operation status according to the touch gesture operation.

[0049] If, in step ST109, the extinguishing area 241 is tapped in the line-of-sight absence state, the operation is considered an operation on the bottom of the screen and the selection value is reduced, since the operation is not the operation of deleting extinguishing area 241.

[0050] The assessment by the line-of-sight attendance / absence assessment unit 12 in step ST103 and the assessment operation by the operation assessment unit 14 in step ST109 can be implemented by storing an operation assessment table such as the one shown in Table 1, for example, in the storage unit of the line-of-sight attendance / absence assessment unit 12 and performing the processing according to the contents of the table. TABLE 1: OPERATIONAL EVALUATION UNIT 14 VISUAL LINE PRESENCE STATE VISUAL LINE ABSENCE STATE VISION ASSESSMENT PRESENCE / ABOUT ASSESSMENT Assess as "absence of sight" condition when push-and-hold operation, continuation of a two-point tactile operation, or continuation of a palm tactile operation. Assess the condition as a "sightline presence state" when a legal strike operation occurs >> Delete >> FULL SCREEN GESTURE - INCREASE VALUE BY 1 WHEN TOP OF SCREEN IS TAPED - DECREASE CHOICE VALUE BY 1 WHEN THE BOTTOM OF THE SCREEN IS TAPED - Finalize the vote if a two-point keystroke is performed and both fingers are removed from the screen within a predetermined time. SCREEN COMPONENTS GESTURE SCROLL LIST WHEN LIST IS DASHED UP OR DOWN - SWIPE SCREEN WHEN SCREEN IS SWIPED LEFT OR RIGHT - DETERMINE VALUE, WINDOW BUTTON IS TAPPED - (1-3) Effect

[0051] As described above in the first embodiment, the method of the operation evaluation unit 14 for evaluating the input information A1, the screen control method of the screen control unit 16, and the notification method of the notification unit 15 can be modified accordingly, depending on whether the operation information A0 indicates a touch gesture operation in the line-of-sight presence state or a touch gesture operation in the line-of-sight absence state, based on the user's touch gesture operation. Thus, while viewing the screen of the touch panel 20, the user can perform a touch gesture operation in the line-of-sight presence state by tapping a button as a screen component displayed on the screen with a finger, swiping a list displayed on the screen with a finger, or the like.When the screen is not being viewed, the user can perform an operation by receiving the touch gesture command using the entire screen and being notified of the operating state. As described above in the first embodiment, regardless of whether the user performs a touch gesture operation on the touchpad 20 while in the line-of-sight state or in the line-of-sight absence state, the touch gesture operation is evaluated accordingly, and thus the input operation can be correctly executed using the touch gesture.

[0052] If, in the first embodiment, an operation that should not be considered a tactile gesture operation in the line-of-sight presence state is performed in the line-of-sight presence state, the operation is accordingly assessed as a tactile gesture operation in the line-of-sight absence state. Conversely, if an operation that should not be considered a tactile gesture operation in the line-of-sight absence state is performed in the line-of-sight absence state, the operation is accordingly assessed as a tactile gesture operation in the line-of-sight presence state. In this case, the accuracy of the assessment across the operation mode increases, since misjudgments are rare in the line-of-sight presence / absence assessment. (1-4) Modifications

[0053] As another example of line-of-sight presence / absence assessment in step ST103 in Fig.4. The line-of-sight presence / absence assessment unit 12 can assess that the operation mode is the operation mode in the line-of-sight absence state if three or more touch points (contact points of fingers) occur. In this case, a two-point touch operation on the touchpad 20 (e.g., an operation such as pinching, rotating, etc.) can be used as the touch gesture operation.

[0054] Furthermore, it serves as another example of line-of-sight presence / absence assessment in step ST103 in Fig.4. It is possible to assess that the state is the line-of-sight absence state if a condition has persisted without any changes to the contents of the touchpad 20 screen for a predetermined time or longer since the start of contact between a finger and the touchpad 20 screen. For example, the line-of-sight presence / absence assessment unit 12 can assess that the state is the line-of-sight absence state if neither screen scrolling by swiping a finger left or right nor list scrolling by swiping the screen up or down occurs, even if an operation of touching the touchpad 20 screen with a finger and swiping the screen has occurred.

[0055] As another example of line-of-sight presence / absence assessment in step ST103 in Fig.4 The line-of-sight presence / absence assessment unit 12 can assess that the line-of-sight presence state is present when movement of the contact point of a finger in a vertical direction, a horizontal direction, or an oblique direction is detected.

[0056] As another example of line-of-sight presence / absence assessment in step ST103 in Fig. 4 can assess the line-of-sight presence / absence assessment unit 12 that the line-of-sight presence state is when a specific location such as a circle, square, or symbol is drawn by moving the contact point of a finger.

[0057] Furthermore, it is a condition for the assessment by the line-of-sight attendance / absence assessment unit 12 in step ST103 in Fig.4. To change from the line-of-sight absence state to the line-of-sight presence state, it is possible to use the elapse of a predetermined time in a state without input on the keypad 20. For example, the device can be configured to automatically return to the line-of-sight presence state if the state without keystroke operation persists for a predetermined threshold time or longer since switching to the line-of-sight absence state. In this case, the user can again perform a keystroke operation, which is judged as an operation in the line-of-sight absence state, such as the press-and-hold operation, with the aid of a voice announcement from the notification unit 15, and then perform a keystroke operation without looking at the screen.

[0058] Furthermore, it is another operational procedure in the line-of-sight state in step ST109 in Fig.4 In addition to the method for increasing / decreasing the selection value by tapping an upper or lower half-area on the screen, it is also possible to display an image like the one in Fig. 5 shown on the screen, in which several buttons for entering a numerical value are arranged, and to cause the notification unit 15 to read aloud the elements of buttons according to positions swiped with a finger.

[0059] As a further operational procedure in the line-of-sight state in step ST109 in Fig.4. It is possible to touch two points on the screen with fingers and rotate the fingers like turning a dial to increase the selection value if the fingers are rotated clockwise and decrease the selection value if the fingers are rotated counterclockwise. If, in this case, two-point tapping is set as a determining operation, there is a possibility that the determining operation and a rotation operation could be confused. Thus, performing two consecutive taps (performing a double tap) within a predetermined time period could be set as a single operation for determining the selection value, for example. (2) Second embodiment (2-1) training

[0060] In the first embodiment described above, the touch gesture operations in operation mode during the line-of-sight state are limited to simple touch gesture operations that utilize the entire screen of the touch field 20 as the operationally effective area. A second embodiment provides a description of a device and a method that prevent unintended operations caused by not viewing the screen and enable more complex touch gesture operations in operation mode during the line-of-sight state.

[0061] Fig. Figure 6 is a functional block diagram showing a schematic configuration of a keypad input device 2 according to the second embodiment of the present invention. Fig. 6 is each component that contributes to a Fig. The component shown in point 1 is identical to or corresponds to it, using the same reference symbol as in point 1. Fig.1. The keypad input device 2 contains the keypad 20 and a key gesture evaluation device 10a. The key gesture evaluation device 10a is a device capable of executing a key gesture evaluation procedure and a key gesture evaluation program according to the second embodiment. The key gesture evaluation device 10a in the second embodiment differs from the key gesture evaluation device 10 in the first embodiment in that it contains a gesture operation storage unit 17 and a gesture correction unit 18. Except for these features, the second embodiment is the same as the first embodiment. Thus, the following description focuses mainly on the difference.

[0062] The gesture operation storage unit 17 stores history information about tactile gesture operations recognized by the operation evaluation unit 14. The gesture correction unit 18 determines a selection value A8 by correcting a tactile gesture operation recognition result as the result of the evaluation by the operation evaluation unit 14 based on the data of the tactile gesture operations stored in the gesture operation storage unit 17.

[0063] A hardware / software configuration of the keypad input device 2 according to the second embodiment includes the keypad 20, the processor 31, the memory 32 and the loudspeaker 33, as shown in Fig. 2 shown. The in Fig.The touch gesture evaluation device 10a shown in Figure 6 can be implemented by using memory 32 as a storage device for storing the touch gesture evaluation program as software and processor 31 as an information processing unit for executing the touch gesture evaluation program stored in memory 32 (e.g., by a computer). In this case, components 11 to 18 correspond to Fig. 6 the processor 31, which runs the touch gesture evaluation program in Fig. 2 executes. Incidentally, it is also possible to execute part of the in Fig. 6 shown tactile gesture assessment device 10a by the in Fig. to implement the memory 32 shown in 2 and the processor 31, which executes the keystroke evaluation program. (2-2) Operation

[0064] Fig.Figure 7 is a flowchart showing the operation of the touch gesture evaluation device 10a (touch gesture evaluation procedure) in the keypad input device 2 according to the second embodiment. Fig. 7 is each process step that involves a Fig. The process step shown in step 4 is identical or corresponds to it, has the same step number as in Fig.4. The operation of the touch gesture evaluation device 10a in the second embodiment differs from the operation of the touch gesture evaluation device 10 in the first embodiment in that a process step for storing a touch gesture operation (step ST201) and a gesture correction process step (step ST202) are executed after the touch gesture operation evaluation step ST109. Except for these features, the operation of the touch gesture evaluation device 10a in the second embodiment is the same as the operation of the touch gesture evaluation device 10 in the first embodiment.

[0065] In step ST201 in Fig. The gesture operation storage unit 17 stores information about a tactile gesture operation recognized by the operation evaluation unit 14 as history information. Table 2 shows an example of the history information for tactile gesture operations. TABLE 2: EXAMPLE OF DATA STORED AS BACKGROUND INFORMATION IN GESTURE OPERATIONS STORAGE UNIT 17 TIME CENTRAL TAST COORDINATES [Pixels] Rotational deviation [pixels] DISTANCE DEVIATION [Pixels] MOVEMENT DISTANCE [Pixels] T1 (X1, Y1) ΔH1 ΔD1 ΔM1 T2 (X2, Y2) ΔH2 ΔD2 ΔM2 T3 (X3, Y3) ΔH3 ΔD3 ΔM3 : : : : : Tk-1 (PAST TIME) (Xk-1, Yk-1) ΔHk-1 ΔDk-1 ΔMk-1 Tk (Present Time) (Xk, Yk) ΔHk ΔDk ΔMk

[0066] Fig. Figure 8 is a diagram showing a method for obtaining the history information about touch gesture operations in the touchpad input device 2 according to the second embodiment. In step ST201, the history information about touch gesture operations, shown in Table 2, is obtained by the Fig. The 8 methods shown are calculated as examples. In the example of Fig.8. A displacement in the direction of rotation, that is, an angular deviation H [pixels], is defined as the length of a perpendicular line Q2 extending from a point P3 touched in the current touch gesture operation to a straight line Q1 connecting two points P1 and P2 touched in the previous touch gesture operation. A distance deviation D [pixels] is defined as a deviation between the current two-point distance (distance between P1 and P3) L2 and the previous two-point distance (distance between P1 and P2) L1. A movement distance M [pixels] is defined as a movement distance between a midpoint I2 between the current two points (between P1 and P3) and a midpoint I1 between the previous two points (between P1 and P2).

[0067] If the user performs a touch gesture operation without looking at the screen and operation information is not entered into the touch gesture evaluation unit 10a as the user's intention, the gesture correction unit 18 corrects the result of the recognition in step ST202 based on the history information stored by the gesture operation storage unit 17.

[0068] Fig. Figure 9 is a diagram showing an example of a touch gesture operation when the user performs the touch gesture operation without looking at the screen of the keypad 20 in the keypad input device 2, and operation information is not entered into the touch gesture evaluation device 10a as the user's intention. In the case of the Fig.In the touch gesture operation shown in 9, the selection value is increased if the user draws circles with a finger clockwise as the first direction of rotation on the screen of the touch field 20, and the selection value is decreased if the user draws circles with a finger counterclockwise as a second direction of rotation opposite to the first direction of rotation.

[0069] If the user draws circles with a finger without looking at the touchpad 20, the positions of the circles shift bit by bit, and the angular deviation H [pixels] per movement distance M [pixels] even changes if the user intends to draw circles. This makes it difficult to adjust the movements to achieve a predetermined degree of operation. Therefore, the gesture correction unit 18 estimates the shapes of the circles using a least squares method and updates the positions of the circles by using touch location information for a period from the present time to a time prior to a predetermined time, during which the gesture operation storage unit 17 accumulates data.After updating the positions of the circles, the gesture correction unit 18 obtains the deviation in the angle by calculating an angle formed by the previous finger contact point, the center position of the circle, and the current finger contact point. Through the above process, the tactile gesture operation of repeatedly drawing circles can be performed without directing the eyes to the screen of the touchpad 20, as shown in [reference]. Fig. 9 shown, to be implemented.

[0070] In addition to updating the position of each circle, if the circle has an elliptical shape, the gesture correction unit 18 can recalculate the coordinate location by making a correction to the transformation of each ellipse into a circle by changing the horizontal and vertical reduction scales of the ellipse and calculating the angle formed by the previous finger contact point, the center position of the circle and the current finger contact point.

[0071] Regarding a period before the gesture correction unit 18 corrects the position of each circle, the operation evaluation unit 14 assesses the tactile gesture operation of drawing circles. The initial position of the circles can in this case either be set as the center position of the screen of the touch field 20 or be determined by a method such as the least squares method based on a location according to a predetermined time.

[0072] Fig. Figure 10 is a diagram showing an example of a touch gesture operation performed without viewing the screen of the touchpad 20 in the touchpad input device 2 according to the second embodiment. An operation used as a trigger to cause the operation evaluation unit 14 to begin evaluating the touch gesture operation of drawing circles can be defined as an operation of touching multiple contact points with multiple fingertips and reducing the number of contacting fingers to below a predetermined number, as shown on the left side of Figure 10. Fig. 10 is shown. For example, if the user touches three points and then the number of contact points is reduced to 1, as in Fig.As shown in Figure 10, the operation evaluation unit 14 can consider the center of gravity position (the center of gravity) of three points as the center and evaluate the tactile gesture operation of drawing circles, and thereafter the correction of the rotation extent can be carried out by updating the position of circles by the gesture correction unit 18 when accumulated history records reach or exceed a predetermined data volume.

[0073] As another example of a touch gesture operation performed by the user without looking at the screen and not as intended by the user, a touch gesture operation of a multi-point pinch, a rotation, and a parallel translation can be given. Fig.Figure 11 is a diagram showing a situation in which the direction, angle and scaling extent of the camera in the keypad input device 2 according to the second embodiment are adjusted without looking at the keypad input device 2 which is at hand. Fig. 11 explains the situation in which the direction, angle and scaling of the camera are adjusted without looking at the touchpad that is at hand.

[0074] In Fig. 11 displays a camera image display unit 401 showing an image captured by the camera. Additionally, the notification unit 15 on the camera image display unit 401 overlays the current camera operation status (direction, scaling factor, rotation angle).

[0075] An available operating screen 402 provides tactile information to the operating information input unit 11, switches the screen display between the line-of-sight presence state and the line-of-sight absence state, and changes the screen content on the screen according to the tactile gesture operation as a result of the assessment by the operating assessment unit 14.

[0076] In the operation mode in the line-of-sight presence state, screen 403, as a screen that is displayed in the case where the line-of-sight presence / absence assessment unit 12 assesses that the state is the line-of-sight presence state, shows a manual on which a touch gesture operation of tapping a hyperlink to switch the screen can be performed.

[0077] In the operation mode in the line-of-sight absence state, screen 404 is a screen that is displayed when the line-of-sight presence / absence assessment unit 12 assesses that the state is the line-of-sight absence state.

[0078] The line-of-sight presence / absence assessment unit 12 assesses that the line-of-sight presence state is present when a three-or-more-point touch operation is detected, a all-finger removal operation from the screen is performed, and a predetermined time has elapsed. The operation assessment unit 14 assesses whether a tactile gesture operation of scaling, moving, or rotating occurs after the three-or-more-point touch operation and displays an image with a change in direction or the degree of camera scaling, such as a scaled camera image 407, a moving camera image 408, or a rotated camera image 409, according to each degree of operation.

[0079] As yet another example of a touch gesture operation performed by the user without looking at the screen and not as intended by the user, there is the operation of hard pressing of the touchpad. For example, if the user performs an operation that the line-of-sight presence / absence assessment unit is supposed to judge as being in the line-of-sight absence state due to the press-and-hold operation, the user who performs the press-and-hold operation without looking at the screen tends to press their finger down hard.

[0080] When the user presses down firmly on the finger, the touch position tends to move slightly from the fingertip to the fingerprint portion of the finger. Therefore, the gesture correction unit 18 makes the assessment of the push-and-hold operation based on the history information on touch gesture operations in Table 2, using either a condition that the movement distance of the finger position does not change beyond a certain distance, or a condition that the movement distance decreases below a certain distance due to a reduction in the pressing pressure.

[0081] The correction process of the gesture correction unit 18 can be applied not only to the push-and-hold operation, but also to tapping and two-point operations.

[0082] If a tactile gesture operation of scaling, moving, or rotating is performed without looking at the screen, there are cases where the tactile gesture operation is incorrectly recognized as a rotation operation, even though the user intended to scale up. Therefore, the gesture correction unit 18 receives the total sum of movement distances from three or more contact points, the total sum of changes or distances between two contact points, and the total sum of changes in the direction of rotation over a period from the present time to a time prior to a predetermined time, accumulated in the gesture operation storage unit 17, and outputs the operation with the largest total sum among these totals of changes as the intended operation. This operation allows the user to perform the intended operation even when performing the tactile gesture operation without looking at the screen. (2-3) Effect

[0083] As described above, in the second embodiment, history records of gestures recognized by the operation evaluation unit 14 are collected and a correction is made, thereby preventing unintended operations caused by operations performed without looking at the screen, and making it possible to perform even more complicated tactile gesture operations without looking at the screen. (3) Third embodiment (3-1) design

[0084] In the first and second embodiments, the line-of-sight presence / absence assessment unit 12 performs the line-of-sight presence / absence assessment (operation mode assessment) based on the operation information A0 according to the touch gesture operation. However, in the keypad input devices 1 and 2 according to the first and second embodiments, the operation used for the line-of-sight presence / absence assessment cannot be used as the touch gesture operation for information input, which is an original purpose.Thus, in the third embodiment, instead of performing the line-of-sight presence / absence assessment (operation mode assessment) based on the operation information A0 according to the touch gesture operation, a camera is provided as an image capture device to capture an image of the user's face, and the line-of-sight presence / absence assessment is performed based on a camera image of the user's face.

[0085] Fig. Figure 12 is a functional block diagram showing a schematic representation of a keypad input device 3 according to the third embodiment. Fig. 12 is each component that contributes to a Fig. The component shown in point 1 is identical to or corresponds to it, using the same reference symbol as in point 1. Fig.1 assigned. The keypad input device 3 includes the keypad 20 and a touch gesture evaluation device 10b. The touch gesture evaluation device 10b is a device that can execute a touch gesture evaluation procedure according to a third embodiment and a touch gesture evaluation program according to the third embodiment. The touch gesture evaluation device 10b in the third embodiment differs from the touch gesture evaluation device 10 in the first embodiment in that a line-of-sight presence / absence evaluation unit 12b performs the line-of-sight presence / absence evaluation (operation mode evaluation) based on a camera image of the user's face, which is received from a camera 34 via a camera image input unit 19. Except for this feature, the third embodiment is the same as the first embodiment.Furthermore, the camera image input unit 19 and the line-of-sight presence / absence assessment unit 12b in the third embodiment can also be applied to the keypad input device 2 according to the second embodiment.

[0086] The camera image input unit 19 receives a camera image (image data) obtained by the camera 34 by capturing an image of the user's face. The line-of-sight presence / absence assessment unit 12b, acting as the operation mode assessment unit, receives the camera image of the user's face from the camera image input unit 19, performs image processing to extract image data of the entire face and eyes, and thereby detects the orientation of the face and the direction of the eyes' line of sight. The operation mode is assessed as the operation mode in the line-of-sight presence state if the eyes are directed toward the screen of the keypad 20, or as the operation mode in the line-of-sight absence state if the eyes are not directed toward the screen of the keypad 20.

[0087] Fig.Figure 13 is a diagram showing an example of a hardware / work configuration of the keypad input device 3 according to the third embodiment. Fig. 13 is each component that contributes to a Fig. The component shown in section 2 is identical or corresponds to it, using the same reference symbol as in the previous section. Fig. 2 assigned. The H / W training of the keypad input device 3 according to the third embodiment differs from the H / W training of the keypad input device 1 according to the first embodiment in that it includes the camera 34 and the touch gesture evaluation program stored in a memory 32b. Except for these features, the H / W training in the third embodiment is the same as that in the first embodiment.

[0088] The camera 34 stores the camera image (image data) of the user's face, which is obtained by the camera's image capture, in memory 32b as the storage unit, and the processor 31 receives the camera image from memory 32b in the process of the camera image input unit 19 and performs image processing in the line-of-sight presence / absence assessment unit 12b. (3-2) Operation

[0089] Fig. Figure 14 is a flowchart showing the operation of the touch gesture evaluation device 10b (touch gesture evaluation procedure) in the touchpad input device 3 according to the third embodiment. Fig. 14 is each process step that involves a Fig. The process step shown in step 4 is identical or corresponds to it, has the same step number as in Fig.4. The operation of the touch gesture evaluation device 10b in the third embodiment differs from the operation of the touch gesture evaluation device 10 in the first embodiment in that the line-of-sight presence / absence evaluation unit 12b performs the line-of-sight presence / absence evaluation based on the camera image of the user's face, which is obtained via the camera image input unit 19 (steps ST301 and ST302), and the operation information A0 according to the touch gesture operation is obtained after the line-of-sight presence / absence evaluation based on the camera image (step ST303). Except for these features, the operation of the keypad input device 3 according to the third embodiment is the same as the operation in the first embodiment. Thus, the following description concerns operations that differ from those in the first embodiment.

[0090] In step ST301 in Fig. 14 The camera image input unit 19 receives the image data of the user's face from the camera 34 and stores the image data in memory 32b.

[0091] The Fig. 15(a) and Fig. Figure 15(b) are diagrams showing a method for assessing the line-of-sight presence and line-of-sight absence states in the keypad input device 3 according to the third embodiment. In step ST302, the line-of-sight presence / absence assessment unit 12b receives the image data of the user's face, as shown in Figure 15(b). Fig. 15(a) shown, stored in memory 32b, by the camera image input unit 19, performs image processing to extract the face direction and the line of sight and judges that the line of sight state is present if the line of sight directed towards the screen of the touch panel 20 is present, as shown on the left side of Fig.15(b) shown. The line-of-sight presence / absence assessment unit 12b assesses that the state is the line-of-sight absence state when the line of sight directed to the screen of the touch panel 20 is missing, as shown on the right side of Fig. 15(b) shown.

[0092] The direction of the face can be detected using a cascade identifier, for example, based on hair feature values, and the line of sight can be detected, for example, by extracting part of the eye as a portion of the face, applying an edge filter to the image of the eye, extracting the outer diameter of the iris, and estimating the position of the pupil by fitting it with an ellipse. The methods for detecting the direction of the face and the line of sight are not limited to these examples, and other methods can be used. (3-3) Effect

[0093] As described above, in the keypad input device 3, the key gesture evaluation device 10b, the key gesture evaluation procedure and the key gesture evaluation program according to the third embodiment, the evaluation as to whether the state is the line-of-sight presence state, in which the user's line of sight is directed at the screen of the keypad 20, or the line-of-sight absence state, as the other state besides the line-of-sight presence state, is carried out by using the camera image from the camera 34, whereby the line-of-sight presence / absence evaluation (operation mode evaluation) can be carried out easily and correctly without the need for the user to perform the key gesture operation to switch the operation mode.

[0094] Furthermore, with the touchpad input device 3, the touch gesture evaluation device 10b, the touch gesture evaluation procedure and the touch gesture evaluation program according to the third embodiment, the line-of-sight presence / absence evaluation is carried out on the basis of the camera image, and thus the limitation when using operations among the touch gesture operations for the original purpose of information input can be eliminated.

[0095] Furthermore, with a mobile terminal device equipped with the keypad input device 3 according to the third embodiment, in cases of testing equipment or viewing a large-area screen device located at a remote position, the user can perform keystroke operations by pressing buttons using ordinary screen components when the user is viewing the keypad 20 (line-of-sight presence state) and use the entire screen of the keypad 20 as an operationally effective area for receiving a keystroke operation when the user is not viewing the keypad 20 (line-of-sight absence state). Thus, when the user is not viewing the keypad 20, the mobile terminal device can be used for testing equipment or as a device for operating a large-area screen device located at a remote position.

[0096] The first to third embodiments described above are an exemplification of the present invention, and a multitude of modifications are possible within the scope of protection of the present invention. DESCRIPTION OF REFERENCE MARKS

[0097] 1, 2, 3: Keypad input unit, 10, 10a, 10b: Key gesture evaluation unit, 11: Operation information input unit, 12, 12b: Line-of-sight presence / absence evaluation unit (operation mode evaluation unit), 13: Operation mode toggle unit, 14: Operation evaluation unit, 15: Notification unit, 16: Screen control unit, 17: Gesture operation storage unit, 18: Gesture correction unit, 19: Camera image input unit, 20: Keypad, 21: Operation field unit, 22: Screen field unit, 31: Processor, 32, 32b: Memory, 33: Speaker, 34: Camera, 141: Screen component gesture evaluation unit, 142: Full screen gesture evaluation unit, A0: Operation information A1: Input information, A2: Line-of-sight assessment information, A3: Command information, A4: Operation assessment information, A5: Notification information, A6: Image signal, A7, A8: Selection value (output signal), A9: Camera image.

Claims

[1] Keypad input device (1, 2), comprising: a keypad (20) that displays an operational image on a screen, receives a keystroke operation performed by a user, and outputs operational information corresponding to the keystroke operation; and a tactile gesture evaluation device (10, 10a) that receives the operation information and generates a choice value based on the operation information, wherein the tactile gesture assessment device (10, 10a) contains: a line-of-sight presence / absence assessment unit (12) that receives input information according to the touch gesture operation, assesses on the basis of the input information whether the touch gesture operation is an operation in a line-of-sight presence state in which the user's line of sight is directed towards the screen, or an operation in a line-of-sight absence state in which the user's line of sight is not directed towards the screen, and outputs line-of-sight assessment information indicating a result of the assessment; an operation mode switching unit (13) that outputs command information based on the line-of-sight assessment information; an operations evaluation unit (14) that generates the selection value based on the operations information through an evaluation procedure according to the command information; and a screen control unit (16) that causes the keypad to display an image according to the command information as the operation image, and the operational evaluation unit (14) contains: a screen component gesture evaluation unit (141) that determines the selection value based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touch panel in the line-of-sight presence state; and a full-screen gesture evaluation unit (142) that determines the selection value based on the operation information according to the touch gesture operation performed on the entire screen of the touch field in the sight-line absence state. [2] Keypad input device according to claim 1, wherein the key gesture evaluation device (10, 10a) further includes a notification unit (15) which provides a notification or outputs a notification signal which has a notification content according to the command information. [3] Touchpad input device according to claim 1 or 2, wherein the line-of-sight presence / absence assessment unit (12) assesses that the user's line of sight is in the line-of-sight absence state when a press-and-hold operation of continuously pressing the touchpad for a time greater than or equal to a predetermined first threshold, or a multi-point operation in which several fingers contact the touchpad, or a palm operation in which a palm contacts the touchpad, has occurred when the touch gesture operation. [4] Keypad input device according to claim 1 or 2, wherein the line-of-sight presence / absence assessment unit (12) assesses that the user's line of sight is in the line-of-sight absence state when an operation that does not change the screen content on the keypad continues for a time greater than or equal to a predetermined second threshold than the keystroke operation. [5] Touchpad input device according to any one of claims 1 to 4, wherein the line-of-sight presence / absence assessment unit (12) assesses that the user's line of sight is in the line-of-sight absence state when a swipe operation is performed over a specified area on the touchpad in a predetermined direction without moving out of the specified area. [6] Touchpad input device according to any one of claims 1 to 5, wherein the line-of-sight presence / absence assessment unit 12) assesses that the user's line of sight is in the line-of-sight absence state when a specific location is drawn on the touchpad screen. [7] Touchpad input device according to any one of claims 1 to 6, wherein the line-of-sight presence / absence assessment unit (12) assesses that the user's line of sight is in the line-of-sight absence state when no operation is performed on the touchpad screen for a time greater than or equal to a predetermined third threshold. [8] Touchpad input device according to any one of claims 1 to 7, wherein, when the line-of-sight presence / absence evaluation unit (12) evaluates that the touch gesture operation is the operation in the line-of-sight absence state, the operation evaluation unit (14) increases the selection value when a predetermined area on a first end side of the touchpad screen is touched, and decreases the selection value when an area on a second end side of the touchpad screen opposite the first end side is touched. [9] Touchpad input device according to any one of claims 1 to 7, wherein, when the line-of-sight presence / absence evaluation unit (12) evaluates that the touch gesture operation is the operation in the line-of-sight absence state, the operation evaluation unit (14) selects a value corresponding to a position where a swipe operation is performed on the touchpad screen as the selection value. [10] Touchpad input device according to any one of claims 1 to 7, wherein, when the line-of-sight presence / absence evaluation unit (12) evaluates that the touch gesture operation is the operation in the line-of-sight absence state, the operation evaluation unit (14) increases the selection value when an operation of rotating multiple contact points on the touchpad screen is performed in a first direction of rotation, and decreases the selection value when an operation of rotating multiple contact points is performed in a second direction of rotation opposite to the first direction of rotation. [11] Touchpad input device according to any one of claims 1 to 7, wherein the touch gesture evaluation device (10a) further comprises: a gesture operation storage unit (17) that stores history information about the tactile gesture operations; and a gesture correction unit (18) that corrects the recognition of the tactile gesture operation by the operation evaluation unit based on the history information stored by the gesture operation storage unit. [12] Touchpad input device according to claim 11, wherein when the line-of-sight presence / absence assessment unit (12) assesses that the touch gesture operation is the operation in the line-of-sight absence state, The operation evaluation unit (14) increases the selection value when an operation of rotating multiple contact points on the touch panel screen is performed in a first direction of rotation, and decreases the selection value when an operation of rotating multiple contact points in a second direction of rotation opposite to the first direction of rotation is performed, and The gesture correction unit corrects the position and location of a circle drawn by the rotation operation. [13] Touchpad input device according to claim 1, wherein an initial position of a contact point in an operation of repeatedly drawing circles by moving the contact point on the screen of the touchpad is a center of gravity or a midpoint of several contact points. [14] Touchpad input device according to claim 11, wherein, when the line-of-sight presence / absence assessment unit (12) assesses that the touch gesture operation is the operation in the line-of-sight absence state, the gesture correction unit Movement distances of three or more contact points entered on the touch panel screen, a scaling level specified by changes in the distance between two contact points, and a rotation angle with respect to a period from a present time to a time before a predetermined time are obtained. obtains a total sum of the movement distances of the contact points, a total sum of the changes in the distance between the contact points, and a total sum of changes in one direction of rotation, and an operation with the largest total sum among total sums of changes is reported as an intended operation. [15] Touchpad input device according to one of claims 8, 9 and 12, wherein, when the line-of-sight presence / absence assessment unit (12) assesses that the touch gesture operation is the operation in the line-of-sight presence state, the operation assessment unit (14) finalizes the selection value when a multi-point touch operation is performed on the touchpad screen. [16] Touchpad input device according to one of claims 10, 13 and 14, wherein, when the line-of-sight presence / absence assessment unit (12) assesses that the touch gesture operation is the operation in the line-of-sight absence state, the operation assessment unit (14) finalizes the selection value when a double tap is made on the touchpad screen. [17] Touchpad input device according to any one of claims 8 to 10, 12 and 14, wherein, when the line-of-sight presence / absence assessment unit (12) assesses that the touch gesture operation is the operation in the line-of-sight absence state, the operation assessment unit (14) finalizes the selection value if no operation is performed on the touchpad screen for a predetermined time. [18] A touch gesture evaluation device (10, 10a) which receives operation information from a touch panel displaying an operation image on a screen, receives a touch gesture operation performed by a user and outputs the operation information according to the touch gesture operation and generates a choice value based on the operation information, wherein the device comprises: a line-of-sight presence / absence assessment unit (12) that receives input information according to the touch gesture operation, assesses on the basis of the input information whether the touch gesture operation is the operation in a line-of-sight presence state in which the user's line of sight is directed towards the screen, or an operation in a line-of-sight absence state in which the user's line of sight is not directed towards the screen, and outputs line-of-sight assessment information indicating a result of the assessment; an operation mode switching unit (13) that outputs command information based on the line-of-sight assessment information; an operations evaluation unit (14) that generates the selection value based on the operations information through an evaluation procedure according to the command information; and a screen control unit (16) which causes the keypad to display an image according to the command information as the operation image, the operational evaluation unit (14) contains: a screen component gesture evaluation unit (141) that determines the selection value based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touch panel in the line-of-sight presence state; and a full-screen gesture evaluation unit (142) that determines the selection value based on the operation information according to the touch gesture operation performed on the entire screen of the touch field in the sight-line absence state. [19] A touch gesture evaluation method of receiving operation information from a touch field (20) displaying an operation image on a screen, receiving a touch gesture operation performed by a user, and outputting the operation information according to the touch gesture operation, and of generating a choice value based on the operation information, wherein the method comprises: a line-of-sight presence / absence assessment step of assessing based on input information according to the touch gesture operation, whether the touch gesture operation is the operation in a line-of-sight presence state where the user's line of sight is directed at the screen, or an operation in a line-of-sight absence state where the user's line of sight is not directed at the screen, and outputting line-of-sight assessment information that indicates a result of the assessment; an operational evaluation step of generating the choice value based on the operational information by means of an evaluation procedure according to command information based on the line-of-sight evaluation information; and a screen control step of causing the keypad to display an image according to the command information as the operation image, wherein in the operation evaluation step the choice value in the line-of-sight presence state is determined based on the operation information according to the touch gesture operation performed on a screen component displayed as the operation image on the touchpad, and the choice value in the line-of-sight absence state is determined based on the operation information according to the touch gesture operation performed on the entire screen of the touchpad. [20] A touch gesture evaluation program for receiving operation information from a touch panel (20) displaying an operation image on a screen, receiving a touch gesture operation performed by a user, and outputting the operation information according to the touch gesture operation, and for generating a choice value based on the operation information, wherein the program causes a computer to execute: a line-of-sight presence / absence assessment process of assessing based on input information according to the touch gesture operation, whether the touch gesture operation is the operation in a line-of-sight presence state where the user's line of sight is directed at the screen, or an operation in a line-of-sight absence state where the user's line of sight is not directed at the screen, and outputting line-of-sight assessment information that indicates a result of the assessment; an operational evaluation process of generating the choice value based on the operational information through an evaluation procedure according to command information based on the line-of-sight evaluation information; and a screen control process of causing the keypad to display an image according to the command information as the operation image, wherein in the operation evaluation process the choice value in the line-of-sight presence state is determined based on the operation information according to the touch gesture operation performed on a screen component that is displayed as the operation image on the touchpad, and in the line-of-sight absence state the choice value is determined based on the operation information according to the touch gesture operation performed on the entire screen of the touchpad.

Images