Operating method

Abstract

To provide an input terminal device that, when detecting the position of a finger in a non-contact manner to perform an input operation, correctly executes processing according to an intended input operation without reacting to an invalid motion of the movement of the finger.SOLUTION: An input terminal device inputting a user's operation through a position input object PS comprises: a position detection unit that detects the position of the position input object operated by the user in a non-contact manner; a display unit that displays a cursor on the basis of the position detected by the position detection unit; and an operation processing control unit that causes corresponding operation processing to be executed on the basis of the position detected by the position detection unit. The operation processing control unit has a plurality of operation modes related to the execution of the operation processing, and one of the operation modes includes a state M0 in which any types of operation processing are not executed other than moving the cursor according to the position of the position input object. The operation processing control unit is configured to switch between the plurality of operation modes when the user performs a specific operation through the position input object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a small input terminal device that is worn on the wrist or the like and an operation input method. [Background technology]

[0002] In recent years, mobile information terminals such as smartphones, which can not only make calls but also perform tasks such as searching for various information, sending and receiving emails, and managing schedules, have become increasingly popular. However, these mobile information terminals are often stored in bags or pockets when traveling, and users must take the mobile information terminal out of their bags or pockets every time they want to check a received email, etc. For this reason, smaller input terminal devices such as wristwatches that connect to these mobile information terminals via wireless communication have been developed.

[0003] While mobile information terminals such as smartphones have touch panels that display information and allow input with the fingers, small input terminal devices have small touch panels, making it difficult to perform fine input operations with the fingers. For this reason, non-contact input methods that use the back of the user's hand as the input surface have been proposed.

[0004] For example, the wristband-type input device described in Patent Document 1 has a light-emitting unit and a light-receiving unit on the main body, and is configured to enable characters to be input by detecting the position of the fingertips on the back or palm of the hand without contact. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-106765 A Summary of the Invention [Problem to be solved by the invention]

[0006] In a conventional character input method using a touch panel or the like, characters are drawn according to the movement of the user's finger when the user's finger is in contact with the panel surface, and characters are not drawn when the user's finger is off the panel surface, so that valid and invalid input periods can be distinguished. As a result, when inputting characters consisting of multiple "strokes" such as line segments and points, the desired character can be correctly input by distinguishing between valid "strokes" and invalid parts that move between "strokes."

[0007] On the other hand, the non-contact input method described in Patent Document 1 detects multiple grid points on the back of the hand where the fingertip passes and recognizes the input character from the resulting dot matrix information. However, the method does not distinguish between valid parts of the strokes that make up the character and invalid parts that move between strokes. This means that invalid parts unrelated to strokes are detected as part of the character, which can easily lead to incorrect recognition of the input character. In other words, the start and end points of character drawing are unclear. Even if these invalid parts are predicted and excluded in advance, incorrect input is likely to be unavoidable when the stroke order is different or when characters containing voiced consonants or many strokes are input, as the trajectory of the invalid parts is not consistent.

[0008] An object of the present invention is to provide an input terminal device that, when detecting the position of a finger in a non-contact manner to perform an input operation, does not react to invalid finger movements and correctly executes processing in accordance with the intended input operation. [Means for solving the problem]

[0009] The present invention provides an input terminal device for inputting user operations via a position input object, comprising a position detection unit that detects the position of the position input object operated by the user in a non-contact manner, a display unit that displays a cursor based on the position detected by the position detection unit, and an operation processing control unit that executes corresponding operation processing based on the position detected by the position detection unit, wherein the operation processing control unit has a plurality of operation modes for executing the operation processing, one of which includes a state in which no operation processing is executed other than moving the cursor according to the position of the position input object, and the operation processing control unit switches between the plurality of operation modes when the user performs a specific operation via the position input object.

[0010] The present invention also provides an operation input method for inputting user operations via a position input object, comprising: a position detection step for contactlessly detecting the position of the position input object operated by the user; a display step for displaying a cursor on a display unit based on the position detected in the position detection step; and an operation processing step for executing corresponding operation processing based on the position detected in the position detection step, wherein the operation processing step has a plurality of operation modes for executing the operation processing, one of which includes a state in which no operation processing is executed other than moving the cursor according to the position of the position input object, and the operation processing step switches between the plurality of operation modes when the user performs a specific operation via the position input object. [Effects of the Invention]

[0011] According to the present invention, an input terminal device can be provided that, when detecting the position of a finger in a non-contact manner to perform an input operation, does not react to invalid finger movements and correctly executes processing in accordance with the intended input operation. [Brief explanation of the drawings]

[0012] [Figure 1A] FIG. 2 is a block diagram showing the configuration of an input terminal device (first embodiment). [Figure 1B] FIG. 2 is a diagram showing an example of a software configuration of an input terminal device. [Figure 2] FIG. 2 is a diagram showing the appearance of an input terminal device. [Figure 3] FIG. 10 is a diagram showing the principle of detecting a finger operation mode. [Figure 4] 10A and 10B are diagrams showing examples of displays on a display unit in a handwritten character input process. [Figure 5] 10 is a flowchart showing operation switching of the entire input terminal device. [Figure 6] 10A to 10C are diagrams showing examples of operation modes and corresponding cursor displays. [Figure 7] 10A and 10B are diagrams showing examples of screen displays in handwritten character input processing. [Figure 8] 8A and 8B are diagrams showing the operation modes and the changes in the drawing position over time in FIG. 7; [Figure 9] 10 is a flowchart showing a handwritten character input process in a finger operation mode. [Figure 10] 10 is a flowchart showing the operation mode determination process S605 in FIG. 9; [Figure 11] FIG. 10 is a diagram showing the configuration of a display unit for key character input processing (Example 2); [Figure 12] 10A and 10B are diagrams showing an example of a key character input process in a finger operation mode. [Figure 13] 10 is a flowchart showing a key character input process in a finger operation mode. [Figure 14] 14 is a flowchart showing the operation mode determination process S1105 in FIG. 13. [Figure 15] 10A to 10C are diagrams showing examples of operation modes and cursor displays used in mouse operation input processing (Embodiment 3); [Figure 16] 10A and 10B are diagrams showing examples of screen displays in mouse operation input processing; [Figure 17] 10 is a flowchart showing a mouse operation input process in a finger operation mode. [Figure 18] 18 is a flowchart showing the operation mode determination process S1504 in FIG. [Figure 19] FIG. 10 is a diagram showing the detection principle of the 3D finger operation mode (Example 4). [Figure 20] FIG. 10 is a diagram showing an example of a screen display in a 3D finger operation mode. [Figure 21] 10 is a flowchart showing a mouse operation input process in a 3D finger operation mode. [Figure 22] 10A to 10C are diagrams showing examples of screen displays in key character input processing. [Figure 23] FIG. 10 is a diagram showing an example of a screen display when a transition area is provided between layers (Example 5). [Figure 24] 10 is a flowchart showing a mouse operation input process when a transition area is provided between layers. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0014] In the first embodiment, the configuration of an input terminal device and a case where handwritten characters are input by detecting a finger position will be described.

[0015] 1A is a block diagram showing the configuration of an input terminal device 1 according to a first embodiment, and the internal configuration (hardware configuration) of the input terminal device 1 will be described. The input terminal device 1 is used by connecting to a mobile information terminal 2 such as a smartphone. The input terminal device 1 of this embodiment has, as operation input functions, a finger operation mode (non-contact detection method) that uses light reflection and a touch operation mode (contact detection method).

[0016] The main control unit 101 is composed of a CPU (Central Processing Unit) and the like, and controls the entire input terminal device 1 in accordance with various operating programs and data stored in a ROM (Read Only Memory) 102 or a RAM (Random Access Memory) 103. The system bus 100 is a data communication path for transmitting and receiving data between the control unit 101 and each unit within the input terminal device 1.

[0017] The ROM 102 is a memory in which various programs for controlling the input terminal device 1 are stored, and uses a rewritable ROM such as an EEPROM (Electrically Erasable Programmable ROM) or a flash ROM. The RAM 103 is used as a temporary storage area during execution of the programs stored in the ROM 102, or as temporary storage of finger position information detected by the finger position detection sensor 106. The storage unit 104 stores information such as operation setting values ​​of the input terminal device 1, and uses a non-volatile rewritable device such as a flash ROM or an SSD (Solid State Drive). The ROM 102 and RAM 103 may be configured integrally with the main control unit 101. Furthermore, the ROM 102 does not have to be an independent configuration, and may use a partial storage area within the storage unit 104.

[0018] The acceleration sensor 105 is a sensor that detects the acceleration of the input terminal device 1. The finger position detection sensor 106 is a sensor that detects the position of a position input object such as a user's finger or a pen in a non-contact manner using reflected light. The short-range wireless communication unit 107 connects to the mobile information terminal 2 by wireless communication and transmits information input to the input terminal device 1. For example, Bluetooth (registered trademark) or WiFi (registered trademark) is used as the communication method.

[0019] The display control unit 108 controls the display of input characters and various setting images of the input terminal device 1 on the display unit 109a. The touch panel 109 includes the display unit 109a and a touch operation input unit 109b. The display unit 109a is a display device such as a liquid crystal panel, and displays characters, images, etc. under the control of the display control unit 108. The touch operation input unit 109b is disposed on the display surface of the display unit 109a.

[0020] The touch operation input unit 109b is, for example, a touchpad-type input device such as a capacitance-type, and detects touch operations using a finger, a touch pen, or the like as operation input. Specifically, it detects, for example, a gesture called a swipe, in which a finger touches the touch panel and then moves the finger in a specific direction while still touching the panel, a gesture called a flick, in which a finger touches the touch panel and then moves the finger at a speed equal to or greater than a predetermined value and then releases the finger, and a gesture called a tap, in which a finger touches the touch panel and then quickly releases the finger. This enables operation input to the input terminal device 1 using the touch method.

[0021] The clock unit 110 measures the elapsed time from the date and time set by the user, for example, using an RTC (Real Time Clock) circuit, and outputs the date and time information. The main control unit 101 performs timer and other processing by reading the date and time information output from the clock unit 110. The operation input unit 111 is an instruction input unit that inputs operation instructions to the input terminal device 1, and has buttons for turning the power on / off and setting the operation mode.

[0022] As described above, the input terminal device 1 of this embodiment is configured to include a finger position detection sensor 106 that detects the position of a user's finger in a non-contact manner to input characters, etc., a touch operation input unit 109b that inputs operations by touching the touch panel 109 with a finger, and a display unit 109a that displays characters, images, etc. This enables two operation input functions: a finger operation mode and a touch operation mode.

[0023] FIG. 1B is a diagram showing an example of the software configuration of the input terminal device 1, and shows the software configuration in the ROM 102, RAM 103, and storage unit 104.

[0024] The main control unit 101 executes a program stored in the ROM 102 to constitute a basic operation processing control unit 1021, a finger position information acquisition unit 1022, and a touch position information acquisition unit 1023. The RAM 103 also includes a temporary storage area 1031 that temporarily stores data as needed when executing a program stored in the ROM 102. The storage unit 104 stores setting information 1041 for use when executing a program stored in the ROM 102. For ease of explanation, the following description will be given assuming that the main control unit 101 controls each operation block by executing a program stored in the ROM 102.

[0025] The basic operation processing control unit 1021 controls various settings and the overall operation of the input terminal device 1. The finger position information acquisition unit 1022 acquires position information of a position input object such as a user's finger or pen based on the output of the finger position detection sensor 106. The touch position information acquisition unit 1023 acquires position information of the position where the finger, touch pen, or the like has come into contact with the touch panel 109 based on the output of the touch operation input unit 109b.

[0026] The finger operation processing control unit 1024 determines what kind of operation has been input based on the information on the position of the position input object acquired from the finger position information acquisition unit 1022, and controls processing corresponding to the determination result. The touch operation processing control unit 1024 determines what kind of operation has been input based on the information on the position of the touch on the touch panel 109 acquired from the touch position information acquisition unit 1023, and controls processing corresponding to the determination result.

[0027] FIG. 2 is a diagram showing the appearance of the input terminal device 1. It is a wristwatch-type input terminal device that is worn on the user's wrist. A touch panel 109 (display unit 109a) is arranged on the surface of the device. Two sensors 106a and 106b are arranged as finger position detection sensor 106 on the side of the device facing the back of the user's hand. These sensors are made of, for example, light-emitting / light-receiving elements. In addition, an operation button is arranged as an operation input unit 111 on the side of the device facing the user's arm.

[0028] FIG. 3 is a diagram illustrating the detection principle of the finger operation mode, showing the relationship between the finger position and the display position on the display unit. The user's finger (position input object) PS moves, for example, within position ABCD on the back of the user's hand. The finger position detection sensors 106a and 106b are composed of an infrared light-emitting element and a light-receiving element. The infrared light emitted from the light-emitting element is reflected by the finger PS, and the reflected light is detected by the light-receiving element. The level of the detection signal from the light-receiving element corresponds to the distance La, Lb from each sensor 106a, 106b to the finger PS.

[0029] The output signal of the finger position detection sensor 106 is sent to the finger position information acquisition unit 1022, which acquires the position F of the finger PS in the position input detection range Pa represented by the position ABCD. The positions ABCD in the position input detection range Pa correspond to the positions abcd in the display range Da of the display unit 109a of the touch panel 109. As a result, the display position f of the cursor or the like in the display range Da is determined in accordance with the position F of the finger PS in the position input detection range Pa.

[0030] 4 is a diagram showing an example of a display on display unit 109a during handwritten character input processing. Area 501 displays the results of recognizing input handwritten characters, and area 502 displays handwritten characters input with a finger. Buttons 503 selectable by the user for character input processing include recognition process instruction button 503a for selecting a process for recognizing handwritten characters, character type switching process instruction button 503b for switching the type of input character, such as hiragana or alphabet, deletion process instruction button 503c for erasing the display of input handwritten characters, character conversion process instruction button 503d for selecting a process for converting the character string of the recognition result displayed in area 501 into kanji or the like, and end process instruction button 503e for terminating the handwritten character input processing.

[0031] 5 is a flowchart showing the operation switching of the entire input terminal device 1. Here, pressing a button on the operation input unit 111 switches the power on / off and the operation input mode (finger operation mode / touch operation mode).

[0032] In S301, the output of the operation input unit 111 is taken in, and in S302, it is determined whether the operation input unit 111 is pressed. If the operation input unit 111 is pressed (Yes), the process branches to S306. In S306, the time that the operation input unit 111 has been pressed is determined, and the process proceeds to power on / off switching or operation input mode switching processing.

[0033] If the operation input unit 111 has been pressed for a predetermined time Tp or more in the determination process S306 (Yes), the process branches to S313, where an on / off switching process is performed depending on whether the power is on or not. If the power is on in S313 (Yes), the process branches to S314, where the power is turned off, and returns to S301. If the power is turned off, the supply of power to components other than the main control unit 101, the timer unit 110, and the operation input unit 111 is stopped in order to reduce power consumption. If the power is off in S313 (No), the process branches to S315, where the power is turned on, and returns to S301. If the power is on, power is supplied to each component constituting the input terminal device 1.

[0034] If the operation input unit 111 has not been pressed for a predetermined time Tp or longer in the determination process S306 (No), the process branches to S307 to determine whether the power is on. If the power is off (No), the process returns to S301. If the power is on (Yes), the process branches to S308.

[0035] In S308, a process of switching the operation input mode is performed. That is, if the current operation input mode is set to the finger operation mode (Yes), in S309, the operation input mode is switched to the touch operation mode, and in S310, a setting is made so that the cursor is not displayed on the display unit 109a of the touch panel 109. If the current operation input mode is not set to the finger operation mode (No) (that is, if it is set to the touch operation mode), in S311, the operation input mode is switched to the finger operation mode, and in S312, a setting is made so that the cursor is displayed on the display unit 109a of the touch panel 109.

[0036] If the operation input unit 111 is not pressed (No) in the determination process S302, the process proceeds to S303, where processing is performed according to the set operation input mode. If the operation mode is finger operation mode (Yes), the process proceeds to S304, where it is determined what type of operation has been input based on the finger position information acquired from the finger position information acquisition unit 1022, and processing corresponding to the determination result is performed. If the operation mode is not finger operation mode (No) (i.e., if the operation mode is touch operation mode), the process proceeds to S305, where it is determined what type of operation has been input based on the touch position information acquired from the touch position information acquisition unit 1023, and processing corresponding to the determination result is performed. After performing the processing in the finger operation execution process S304 or the touch operation execution process S305, the process returns to S301.

[0037] The handwritten character input process in the finger operation mode will be specifically described below. In this embodiment, multiple operation states (operation modes) are provided for finger operation, so that handwritten character input can be performed correctly and easily.

[0038] FIG. 6 is a diagram showing an example of operation modes and the corresponding cursor display. Operation mode M0 is a state in which no drawing for character input other than cursor movement is performed even when a finger is operated (drawing is stopped). Operation mode M1 is a state in which drawing for character input is performed according to the position of a finger when the finger is operated. Also, the cursor display (for example, white / black display) is switched according to the operation mode so that the user can easily identify the current operation mode. Note that, as will be described later, the operation mode can be switched from M0 to M1, or conversely, from M1 to M0, by performing a specific operation in which the operating finger is held at a certain position on the operation surface for a predetermined time t0 or more.

[0039] As a result, drawing is only possible when the operation mode is M1. Therefore, when inputting characters consisting of multiple "strokes" such as line segments and dots, by setting the operation mode to M1 for the valid "strokes" and setting the operation mode to M0 for the invalid parts that move between "strokes," the desired character can be input easily and correctly.

[0040] 7 is a diagram showing an example of a screen display in handwritten character input processing. Here, inputting the hiragana character "a" is used as an example. The character input procedure is shown as (S1) to (S9), and the operation mode (M0 / M1) at that time is also indicated.

[0041] In the following explanation, the cursor is said to be "held" at a certain position, but even if the cursor position moves (shakes) within a certain distance r0, this is tolerated and considered to be in the "held" state. This r0 is due to unintentional movement by the user (such as hand shake), and is set to a value of, for example, several mm. Furthermore, the time t0 required to determine the "held" state is set to a value of, for example, several seconds. These thresholds may be set according to the user and the usage environment.

[0042] (S1): When the operation mode is M0 (drawing stopped), the finger PS is moved within the position input detection range Pa to move the cursor to the position 7a in the input / display area 502 for characters. (S2): The cursor is held at position 7a for a predetermined time t0 or more, which switches the operation mode to M1 (drawing execution) and changes the cursor display.

[0043] (S3): When the cursor display changes, move your finger to move the cursor to position 7b in the input / display area 502 for characters. This draws a line from position 7a to position 7b. (S4): The cursor is held at position 7b for a predetermined time t0 or more, which switches the operation mode to M0 (drawing stopped) and changes the cursor display.

[0044] (S5): When the cursor display changes, move your finger to move the cursor to position 7c in the input / display area 502 for characters. (S6): Position 7c is maintained for a predetermined time t0 or more, which switches the operation mode to M1 and changes the cursor display.

[0045] The drawing operation continues in the same manner. That is, when drawing, the operation mode is set to M1 and the cursor is moved, and when not drawing, the operation mode is set to M0 and the cursor is moved. Also, by holding the cursor in the moved position for a predetermined time t0 or more, the operation mode is switched and handwritten characters are input.

[0046] (S7): When the drawing of the hiragana character "a" is completed, it is held at position 7d for a predetermined time t0 or more. This switches the operation mode to M0 and changes the cursor display. (S8): The cursor is moved to the recognition processing instruction button 503a. (S9): When the cursor is held on the recognition process instruction button 503a for a predetermined time t0 or more, the recognition process is executed, and the recognition result is displayed in the display area 501.

[0047] FIG. 8 is a diagram showing the change over time in the operation mode and drawing position in FIG. 7. (a) shows the position of the position input object (user's finger) PS, (b) shows the operation mode, and (c) shows the drawing position on the display unit. (S1) to (S6) on the horizontal axis and symbols (7a) to (7c) correspond to the symbols in FIG. 7. In this way, drawing is performed only during the period when the operation mode is M1, and the operation mode is switched by holding the position input object PS in the same position for time t0 or longer.

[0048] 9 is a flowchart showing the handwritten character input process in the finger operation mode. The contents of each process will be explained below in order.

[0049] In S601, the position F of the finger PS is acquired by the finger position information acquisition unit 1022, and in S602, the acquired position F of the finger PS is converted into a position f in the display range Da as described in Fig. 3. Next, in S603, it is determined whether the distance r between the position f and the previously acquired position fd stored in the temporary storage area of ​​the RAM 103 is equal to or less than a predetermined distance r0.

[0050] In the determination process S603, if the distance r is equal to or greater than the predetermined distance r0 (No), the process branches to S617. In S617, it is determined whether or not the position f is within the display range of the character input / display area 502. If the position f is within the range of the character input / display area 502 (Yes), the process branches to S618. If the position f is outside the range of the character input / display area 502 (No), the process branches to S620. In S620, the operation mode is set to M0, and the process proceeds to S615.

[0051] In S618, the set operation mode is determined. If the operation mode is M0, the drawing process is not performed and the process proceeds to S615. If the operation mode is M1, the process branches to S619, where drawing process is performed from position fd to position f in the display range Da, and the process proceeds to S615.

[0052] On the other hand, in the determination process S603, if the distance r is equal to or less than the predetermined distance r0 (Yes), the process branches to S604. In S604, it is determined whether the position f is within the display range of the predetermined area 502 for character input in the display range Da, or within the display range of the processing instruction button 503. If the position f is within the display range of the character input / display area 502 or the processing instruction button 503 (Yes), the process proceeds to S605. If the position f is outside the display range of the character input / display area 502 or the processing instruction button 503 (No), the process proceeds to S615.

[0053] In S605, the operation mode is determined using a timer. That is, the holding time is compared with a predetermined time t0 and the operation mode is switched. Details of the determination process in S605 will be explained in FIG. 10. In S606, branching processing is performed depending on the determined operation mode. If the operation mode is M0, the process proceeds to S615, and if the operation mode is M1, the process proceeds to S607. In S607, it is determined whether or not position f is within the display range of the processing instruction button 503. If position f is within the display range of the processing instruction button 503 (Yes), the process proceeds to S608, and if position f is outside the display range of the processing instruction button 503 (No), the process proceeds to S615.

[0054] In S608, branching processing is performed depending on which of the display ranges of processing instruction buttons 503a to 503e position f is within. If position f is within the display range of recognition processing instruction button 503a, processing proceeds to S609, where recognition processing is performed. If position f is within the display range of character type switching processing instruction button 503b, processing proceeds to S610, where character type switching processing is performed. If position f is within the display range of deletion processing instruction button 503c, processing proceeds to S611, where deletion processing is performed. If position f is within the display range of conversion processing instruction button 503d, processing proceeds to S612, where conversion processing is performed. After each process is executed, the process proceeds to S613, where the timer is reset and stopped, and in S614 the operation mode is set to M0. If the position f is within the display range of the end process instruction button 503e, the handwritten character input process is ended.

[0055] In S615, depending on the set operation mode, a cursor is displayed at position f in the display range Da as shown in Fig. 6. Next, in S616, the current position f is replaced with the previous position fd, which is stored in a temporary storage area of ​​the RAM 103, and the process returns to S601.

[0056] FIG. 10 is a flowchart showing the operation mode determination process S605 in FIG.

[0057] In S701, it is determined whether the timer is running. If the timer is not running (No), the process proceeds to S702, where the timer is started and the process ends. If the timer is running (Yes), the process proceeds to S703.

[0058] In S703, it is determined whether the timer has exceeded a predetermined time t0. If the predetermined time t0 has not been exceeded (No), the process ends. If the predetermined time t0 has been exceeded (Yes), the process proceeds to S704.

[0059] In S704, branching processing is performed depending on the operating mode. If the operating mode is M0, the process proceeds to S705, where the operating mode is switched to M1 and set. If the operating mode is M1, the process proceeds to S706, where the operating mode is switched to M0 and set. Thereafter, the process proceeds to S707, where the timer is reset and stopped, and the process ends.

[0060] As described above, in the first embodiment, the operation mode is switched by holding the position of a position input object, such as a user's finger, on the display screen within a predetermined distance r0 for a predetermined time t0 or more. If drawing is to be performed, the operation mode is set to M1 and the cursor is moved, and if drawing is not to be performed, the operation mode is set to M0 and the cursor is moved. This makes it possible to distinguish the movement between strokes as a non-drawing section even when inputting characters with multiple strokes, and to easily and correctly input the intended character even if the stroke order is incorrect. It also makes it possible to input more complex characters, such as kanji.

[0061] In addition, by moving the cursor to the area where the processing instruction button is displayed and holding the cursor within a predetermined distance r0 from the position to which the cursor was moved for a predetermined time t0 or more, the processing corresponding to the processing instruction button can be performed, and the same operation can be performed as by moving the cursor to the position where the processing instruction button is displayed using an input means such as a mouse and clicking it.

[0062] In this embodiment, an example is shown in which the processing instruction buttons are configured as a recognition processing instruction button 503a, a character type switching processing instruction button 503b, an erasure processing instruction button 503c, a character conversion processing instruction button 503d, and an end processing instruction button 503e, but this is not limited to this and other processing instruction buttons may be provided.

[0063] Furthermore, in this embodiment, the operation input mode (touch operation mode) using the touch panel 109 and the operation input mode (finger operation mode) using the finger position detection sensor 106 are switched by the operation input unit 111, but the switching may be performed by other methods. For example, when the back of the hand is tapped with a finger, the acceleration sensor 105 detects vibration, but the touch panel 109 does not detect touch. Therefore, it may be determined that the back of the hand has been tapped based on the detection results of the touch panel 109 and the acceleration sensor 105, and the operation input mode may be switched when the back of the hand is tapped with a finger.

[0064] In this embodiment, the specific operation for switching the operation mode is to hold the position of a position input object, such as a user's finger, on the display screen within a predetermined distance r0 for a predetermined time t0 or more, but this is not limiting. As another specific operation, for example, the operation mode may be switched by moving the cursor by moving the position input object and tapping the back of the hand at the moved position. [Example]

[0065] In the second embodiment, a case where input is performed by selecting keys displayed on a screen like a keyboard will be described. In a small portable information terminal, the keys displayed on the display unit 109a are small, making it difficult to input characters by touching the touch panel 109 with a finger. Therefore, key character input is performed in a finger operation mode.

[0066] 11 is a diagram showing the configuration of display unit 109a for key character input processing. Area 1001 is an area where key character input characters are displayed, and character input keys 1002 are keys for selecting characters to input. Keys 1003 are used to instruct various processes, and include character type switching process instruction key 1003a for switching the type of character to be input, such as hiragana or alphabet, deletion process instruction key 1003b for deleting input characters displayed in area 1001, and termination process instruction key 1003c for terminating key character input processing.

[0067] 12 is a diagram showing an example of a key character input process in the finger operation mode, in which the case of inputting the alphabet "Y" is taken as an example. The input procedure will be explained in (S1) to (S4).

[0068] (S1): In the operating mode M0, the finger PS is moved within the position input detection range Pa to move the cursor to the position 12a where the character input keys "WXYZ" are displayed. (S2): The cursor is held at position 12a for a predetermined time t0 or more (in this case, movement within a predetermined distance r0 is allowed. The same applies below.) This switches the operation mode to M1, and a display appears allowing you to select the letters "W," "X," "Y," or "Z."

[0069] (S3): When the character selection display is displayed, the user moves his / her finger to move the cursor to the position 12b where the character "Y" is displayed. (S4): The cursor is held at position 12b for a predetermined time t0 or more. As a result, the character "Y" is determined as the input character, and the character "Y" is displayed in the input character display area 1001. When the character input is completed, the operation mode switches to M0.

[0070] FIG. 13 is a flowchart showing the key character input process in the finger operation mode.

[0071] In S1101, the position F of the finger PS is acquired by the finger position information acquisition unit 1022, and in S1102, the acquired position F of the finger PS is converted into a position f in the display range Da. Next, in S1103, it is determined whether the distance r between the position f and the previously acquired position fd is equal to or less than a predetermined distance r0.

[0072] In the determination process S1103, if the distance r is equal to or greater than the predetermined distance r0 (No), the process branches to S1114. If the distance r is equal to or less than the predetermined distance r0 (Yes), the process branches to S1104. In S1104, it is determined whether or not the position f is within the display area of ​​the character input keys 1002 or the process instruction keys 1003. In the determination process S1104, if the position f is within the display area of ​​the character input keys 1002 or the process instruction keys 1003 (Yes), the process branches to S1105, and if it is outside the display area of ​​the character input keys 1002 or the process instruction keys 1003 (No), the process branches to S1114.

[0073] In S1105, the operation mode is determined by a timer. That is, the hold time is compared with a predetermined time t0 and the operation mode is switched. Details of the determination process in S1105 will be explained in FIG. 14. In S1106, branching processing is performed depending on the determined operation mode. If the operation mode is M0, the process proceeds to S1114, and if the operation mode is M1, the process proceeds to S1107. In S1107, it is determined whether the character selection display is on or not.

[0074] In the determination process S1107, if the character selection display is on (Yes), the process proceeds to S1116, and if the character selection display is off (No), the process proceeds to S1108. In S1116, it is determined whether or not position f is within the character selection display area. If position f is within the character selection area (Yes), the process proceeds to S1117, and if position f is outside the character selection display area (No), the process proceeds to S1114. In S1117, the character displayed at position f is determined to be the input character and is displayed in the input character display area 1001. In S1118, the character selection display is turned off, and the process proceeds to S1112.

[0075] In S1108, branching processing is performed depending on whether position f is within the display area of ​​either the character input keys 1002 or the process instruction keys 1003. If position f is within the display area of ​​the character input keys 1002, the flow proceeds to S1109, where the character selection display is turned on and the character corresponding to the key displayed at position f is selected and displayed. If position f is within the display area of ​​the character type switching process instruction key 1003a, the flow proceeds to S1110, where character type switching processing is performed. If position f is within the display area of ​​the deletion process instruction key 1003b, the flow proceeds to S1111, where deletion processing is performed. After each processing is performed, the flow proceeds to S1112, where the timer is reset and stopped, and in S1113, the operation mode is set to M0. If position f is within the display area of ​​the termination process instruction key 1003c, the key character input processing is terminated.

[0076] In S1114, depending on the set operation mode, a cursor is displayed at position f in the display range Da as shown in Fig. 6. Next, in S1115, the current position f is replaced with the previous position fd, which is stored in a temporary storage area of ​​RAM 103, and the process returns to S1101.

[0077] FIG. 14 is a flowchart showing the operation mode determination process S1105 in FIG.

[0078] In S1201, it is determined whether or not the timer is running. If the timer is not running (No), the process proceeds to S1202, where the timer is started and the process ends. If the timer is running (Yes), the process proceeds to S1203.

[0079] In S1203, it is determined whether the timer has exceeded a predetermined time t0. If the predetermined time t0 has not been exceeded (No), the process ends. If the predetermined time t0 has been exceeded (Yes), the process proceeds to S1204.

[0080] In S1204, the operation mode is switched to M1. In this case, the operation mode is switched to M0 after key character input is executed in S1113 of Fig. 13, so there is no need to switch from M1 to M0. Then, the process proceeds to S1205, where the timer is reset and stopped, and the process ends.

[0081] As described above, in the second embodiment, it is possible to easily select characters to input or instruct a process to be executed by moving the cursor to a position where a character input key or a process instruction key is displayed and holding the cursor at a position within a predetermined distance r0 from the moved cursor position for a predetermined time t0 or more. This allows the same operation as moving the cursor to the position where a character input key or a process instruction key is displayed using an input means such as a mouse and clicking it.

[0082] In this embodiment, the process instruction keys are configured as the character type switching process instruction key 1003a, the delete process instruction key 1003b, and the end process instruction key 1003c, but the present invention is not limited to this and keys corresponding to other process instructions may be provided. Furthermore, as for the character input keys, a keyboard in which each alphabet or hiragana character corresponds to one key, or a keyboard corresponding to the input of numbers may be displayed and used for input. [Example]

[0083] In the third embodiment, operations such as clicking, double-clicking, and drag-and-drop using a mouse (hereinafter referred to as mouse operation input processing) are realized in the finger operation mode. To accommodate this, the number of types of operation modes has been increased in this embodiment.

[0084] FIG. 15 is a diagram showing examples of operation modes and cursor displays used in mouse operation input processing. Six operation modes M0 to M5 are used here. Operation mode M0 is a state in which no input operation other than cursor movement is performed even when a finger is operated. Operation modes M1 to M5 are states in which mouse operations of click processing (M1), double click processing (M2), start of drag processing (M3), end of drag processing (M4), and drop processing (M5) are performed, respectively. Furthermore, the cursor display (shape and / or color display) is switched depending on each operation mode. Note that the operation mode is switched by comparing the length of time that the finger is held in a certain position on the operation surface during operation with three thresholds t0, t1, and t2, as will be described later.

[0085] 16 is a diagram showing an example of a screen display in mouse operation input processing. A group of icons 1401 corresponding to applications A to D is displayed as a window screen (home screen) on display unit 109a. The user can select a desired application by clicking an icon, execute the application by double-clicking, and move the display position of the icon by dragging and dropping. The flow of the series of processes will be explained in (S1) to (S7).

[0086] (S1): In the operating mode M0, the cursor is moved to the position 14a where the icon [D] is displayed by moving the finger within the position input detection range. (S2): The cursor is held at position 14a for a predetermined time (a range of t0 to t1). This switches the operation mode to M1 (click processing), and icon "D" is selected as the mouse operation target. The cursor display also changes to a display corresponding to M1 in FIG. 15. After that, by moving the finger arbitrarily (in this case, moving the cursor to a position that is a predetermined distance r0 or more away from position 14a) before the predetermined time t1 has elapsed, a click processing is executed on the selected icon "D."

[0087] (S3): When the cursor is held at position 14a for a predetermined time t1 or more, the operation mode switches to M2 (double-click processing), and the cursor changes to a display corresponding to M2 shown in Fig. 15. After that, by moving the finger arbitrarily (moving the cursor a predetermined distance r0 or more from position 14a) before the predetermined time t2 has elapsed, a double-click processing is executed on the selected icon "D."

[0088] (S4): When the cursor is held at position 14a for a predetermined time t2 or more, the operation mode switches to M3 (start of drag processing), and the cursor changes to a display corresponding to M3 shown in FIG. (S5): After that, by moving the finger and moving the cursor from position 14a to position 14b that is at least a predetermined distance r0 away, the operation mode switches to M4 (end of drag process). As a result, the cursor changes to a display corresponding to M4 shown in FIG. 15. Also, icon "D", which was selected as the mouse operation target, moves and is displayed at position 14b where the cursor was moved, and a drag process is executed to move the position of the selected icon.

[0089] (S6): The user further moves the finger to move the cursor and the selected icon [D] to position 14c, and holds the cursor at position 14c for a predetermined time t0 or more. This switches the operation mode to M5 (drop process), and the drop process is executed at position 14c. (S7): The display position of the selected icon "D" is set to 14c by the drop process. After that, the operation mode is switched to M0, and the cursor is displayed according to M0.

[0090] FIG. 17 is a flowchart showing mouse operation input processing in the finger operation mode.

[0091] In S1501, the position F of the finger PS is acquired by the finger position information acquisition unit 1022, and in S1502, the acquired position F of the finger PS is converted into a position f in the display range Da. Next, in S1503, it is determined whether the distance r between the position f and the previously acquired position fd is equal to or less than a predetermined distance r0.

[0092] In the determination process S1503, if the distance r is equal to or less than the predetermined distance r0 (Yes), the process branches to S1504, where the operation mode is determined by a timer. That is, the holding time is compared with the predetermined times t0, t1, and t2 to switch the operation mode. Details of the determination process in S1504 will be explained in FIG. 18. Thereafter, in S1505, it is determined whether the operation mode determined in S1504 is M5 or not. If the operation mode is M5 (Yes), the process branches to S1508, and if the operation mode is not M5 (No), the process branches to S1506. In S1508, a drag operation is performed to set the display position of the icon selected by the drag process to position f. Drop processing is executed, and the process proceeds to S1509.

[0093] In S1506, it is determined whether or not position f is within the icon display area. If position f is within the icon display area (Yes), the process branches to S1507, and if it is outside the icon display area (No), the process branches to S1509. In S1507, the icon for which position f is within the display area is selected as the object to be operated with the mouse, and the process proceeds to S1518. In S1509, the operation mode is set to M0, and the process proceeds to S1510. In S1510, the timer is reset and stopped, and the process proceeds to S1518.

[0094] On the other hand, in the determination process S1503, if the distance r is equal to or greater than the predetermined distance r0 (No), the process branches to S1511. In S1511, the following branching process is performed depending on the operation mode. If the operation mode is M0, no processing is performed and the process proceeds to S1518. If the operation mode is M1, the process proceeds to S1515, where a click process is performed on the icon selected as the mouse operation target in S1507, and the process proceeds to S1516. If the operation mode is M2, the process proceeds to S1514, where a double-click process is performed on the icon selected in S1507 as the object of mouse operation, and the process proceeds to S1516. If the operation mode is M3, the process proceeds to S1513, where the operation mode is set to M4, and the process proceeds to S1517. If the operation mode is M4, the process proceeds to S1512, where the icon selected in S1507 as the mouse operation target is displayed at position f. This performs a drag process to move the position of the selected icon, and then the process proceeds to S1518.

[0095] In S1516, the operation mode is set to M0, and in S1517, the timer is reset and stopped, and the process proceeds to S1518. In S1518, a cursor is displayed at position f in the display range Da as shown in Fig. 15 according to the current operation mode. Next, in S1519, the current position f is replaced with the previous position fd, which is stored in a temporary storage area of ​​RAM 103, and the process returns to S1501.

[0096] FIG. 18 is a flowchart showing the determination process S1504 of the operation mode in FIG. 17.

[0097] In S1601, it is determined whether the timer is activated. If the timer is not activated (No), the process proceeds to S1602 to activate the timer and the process ends. If the timer is activated (Yes), the process proceeds to S1603.

[0098] In S1603, it is determined whether the operation mode is M4. If the operation mode is M4 (Yes), the process proceeds to S1604. In S1604, it is determined whether the timer has exceeded a predetermined time t0. If it has not exceeded the predetermined time t0 (No), the process ends. If it has exceeded the predetermined time t0 (Yes), the process proceeds to S1605 to set the operation mode to M5 and the process ends. In the determination process S1603, if the operation mode is not M4 (No), the process proceeds to S1606.

[0099] In S1606, the following branch process is performed according to the elapsed time of the timer. The following time thresholds have the relationship of t0 < t1 < t2. If the timer has not exceeded the predetermined time t0 (timer ≤ t0), the process ends. If the timer is between the predetermined times t0 and t1 (t0 < timer ≤ t1), the process proceeds to S1609 to set the operation mode to M1 and the process ends. If the timer is between the predetermined times t1 and t2 (t1 < timer ≤ t2), the process proceeds to S1608 to set the operation mode to M2 and the process ends. If the timer has exceeded the predetermined time t2 (timer > t2), the process proceeds to S1607 to set the operation mode to M3 and the process ends.

[0100] As described above, in the third embodiment, the operation mode is switched between M0 and M5 depending on the length of time that the cursor is held within a predetermined distance r0 from the moved position. After the operation mode is switched, by moving the cursor to a position at least the predetermined distance r0, like a flick operation in a touch operation on a smartphone, it becomes possible to realize operations such as clicking, double-clicking, and drag-and-drop with a mouse in the finger operation input mode. Furthermore, the change in the cursor display allows the user to confirm that the operation mode has been switched, making it possible to easily perform the mouse operation as intended by the user. [Example]

[0101] In the fourth embodiment, a case where the position of a user's finger is detected three-dimensionally to perform an input operation (hereinafter referred to as a 3D finger operation mode) will be described.

[0102] 19 is a diagram illustrating the detection principle of the 3D finger operation mode, showing the relationship between the finger position and the display position of the display unit. The position of the finger PS is detected not only in directions parallel to the display unit 109a of the input terminal device 1 (XY axis directions) but also in a direction perpendicular to the display unit 109a (height Z axis direction), and input operation is performed. The diagram shows the relationship between the position F of the detected finger (position input object) PS and the display position f of the display unit 109a.

[0103] Three finger position detection sensors 106a, 106b, and 106c are provided to detect the finger position three-dimensionally. Each of the sensors 106a to 106c is composed of, for example, an infrared light-emitting element and a light-receiving element, and outputs a signal with a level corresponding to the distance La, Lb, or Lc to the finger PS. The finger position information acquisition unit 1022 acquires the position F of the finger PS in the position input detection range (space) Pa surrounded by the positions ABCD-A'-B'-C'-D' based on the output signals from each of the sensors 106a to 106c. The position f in the display range Da surrounded by the positions abcd on the display unit 109a is determined in accordance with the position of the acquired position F in the X and Y axes, and a cursor, for example, is displayed. Meanwhile, multiple display screens are switched as described below in accordance with the position of the acquired position F in the Z axis direction.

[0104] Fig. 20 is a diagram showing an example of a screen display in the 3D finger operation mode. Here, a window screen (home screen) on which icons are arranged is displayed as an operation screen. The height Z direction position relative to the display unit 109a is divided into a plurality of layers L0, L1, and L2, with Z0 to Z3 as boundaries. The window screens are switched and displayed depending on which layer the finger PS is located on. The display states in each layer are indicated by (S1) to (S3).

[0105] (S1): When the position F0 of the finger PS is in the range of Z0 to Z1 in the height direction (layer L0), a group of icons 1901 corresponding to applications A to D is displayed on the window screen. (S2): When the position F1 of the finger PS is in the range of Z1 to Z2 in the height direction (layer L1), a group of icons 1902 corresponding to applications E to H is displayed on the window screen. (S3): When the position F2 of the finger PS is in the range of Z2 to Z3 in the height direction (layer L2), a group of icons 1903 corresponding to applications I to L is displayed on the window screen.

[0106] Thereafter, the mouse operation input process described in Example 3 is executed. That is, an application is selected by clicking on a displayed icon, an application is executed by double-clicking on the icon, and the display position of the icon is moved by dragging and dropping the icon.

[0107] In this way, according to the 3D finger operation mode, the screen display for scanning can be switched for each of a plurality of layers, so that mouse operation input processing can be easily performed for a large number of applications (icons).

[0108] 21 is a flowchart showing a mouse operation input process in the 3D finger operation mode. The same processes as those in the third embodiment (FIG. 17) are denoted by the same reference numerals, and repeated explanations will be omitted.

[0109] In S1501, the finger position information acquisition unit 1022 acquires the three-dimensional position F of the finger PS, and in S1502, the acquired position F in the X and Y directions is converted to position f in the display range Da as described in Fig. 19. Next, in S1520, it is determined which layer range the acquired position F in the Z direction falls within. In S1521, it is determined whether the determined layer LN is the same as the previously determined layer LNd stored in the temporary storage area of ​​RAM 103. If the result of the determination is that the determined layer LN is the same as the previously determined layer LNd (Yes), the process proceeds to S1503. The process from S1503 onwards is the same as in Fig. 17, and the description will be omitted. As a result of the determination, if the determined layer LN is different from the previously determined layer LNd (No), the process proceeds to S1522.

[0110] In S1522, it is determined whether the operation mode is M3 or M4. If the operation mode is M3 or M4 (Yes), the process proceeds to S1523, and if the operation mode is not M3 or M4 (No), the process proceeds to S1525. In S1525, the operation mode is set to M0, and the process proceeds to S1526. In S1526, the timer is reset and stopped, and the process proceeds to S1523.

[0111] In S1523, the determined layer LN is replaced with the previously determined layer LNd and stored in a temporary storage area of ​​RAM 103, and the process proceeds to S1524. In S1524, the screen of display unit 109a is updated to display a window screen (home screen) corresponding to the determined layer LN, and the process proceeds to S1511. The process from S1511 onwards is mouse operation input processing according to each operation mode M0 to M4, but is the same as in Fig. 17, and therefore description thereof will be omitted.

[0112] As described above, by detecting the 3D position of the user's finger and performing operation input, it is possible to divide the device into multiple layers according to the height Z position and display a window screen (home screen) for operation corresponding to each layer. It is also possible to perform mouse operations such as clicking, double-clicking, and drag-and-drop on icons displayed in the window of each layer.

[0113] In the above example, icons corresponding to different applications are displayed on the window screen for each of a plurality of layers, but the present invention is not limited to this, and it is possible to display different operation screens for each layer.

[0114] For example, Fig. 22 is a diagram showing an example of a screen display in the key character input process. In the key character input process, a keyboard with different character types for each layer is displayed on the display unit.

[0115] (S1): When the position F0 of the finger PS is within the range of the layer L0, a keyboard for inputting alphabets and symbols is displayed. (S2): If the position F1 is within the range of the layer L1, a keyboard for inputting hiragana characters is displayed. (S3): If position F2 is within the range of layer L2, a keyboard for entering numbers is displayed. This allows the user to easily change the type of characters to be input by changing the position of the finger PS in the height Z direction.

[0116] As described above, according to the fourth embodiment, the screen display and operation contents can be switched for each of the multiple layers in the 3D finger operation mode, so that various types of operations can be easily realized. Note that the number of layers may be set by the user as appropriate depending on the type of operation to be used. [Example]

[0117] In the fifth embodiment, a configuration will be described in which transition areas are provided between layers in order to smoothly switch between a plurality of layers in the 3D finger operation mode.

[0118] 23 is a diagram showing an example of a screen display when a transition area is provided between layers. The position of the finger PS in the height Z direction is divided into multiple layers L0 to L2, and layer transition areas T01, T12 are provided at the boundaries of each layer. As in the fourth embodiment, different operation window screens (here, a group of icons) are displayed corresponding to each of the layers L0 to L2, but in the layer transition areas T01, T12, a window screen corresponding to one of the adjacent layers is displayed. Below, we will explain how the screen display changes along with the movement of the finger PS in the height direction.

[0119] First, a case where the position of the finger PS is moved from F0 to F4 (that is, from layer L0 to L2) will be described in (S1) to (S5).

[0120] (S1): The position F0 of the finger PS is within the range of the layer L0, and a group of icons 2101 corresponding to applications A to D is displayed on the window screen. (S2): When the position F1 of the finger PS moves to the layer transition area T01 between the layers L0 and L1, the window screen (icon group 2101) displayed on the previous layer L0 continues to be displayed. At that time, the screen background is changed to the display 109T indicating the transition area.

[0121] (S3): Position F2 of finger PS is within the range of layer L1, and icon group 2102 corresponding to applications E to H is displayed on the window screen in a switching manner. (S4): When the position F3 of the finger PS moves to the layer transition area T12 between the layers L1 and L2, the window screen (icon group 2102) displayed on the previous layer L1 continues to be displayed. At that time, the screen background is changed to the display 109T indicating the transition area.

[0122] (S5): Position F4 of finger PS is within the range of layer L2, and icon group 2103 corresponding to applications I to L is displayed on the window screen in a switched manner.

[0123] On the other hand, a case where the position of the finger PS is moved from F4 to F0 (that is, from layer L2 to L0) will be described in (S5') to (S1').

[0124] (S5'): Position F4 of finger PS is within the range of layer L2, and a group of icons 2103 corresponding to applications I to L is displayed on the window screen. (S4'): When the position F3 of the finger PS moves to the layer transition area T12 between the layers L1 and L2, the window screen (icon group 2103) displayed on the previous layer L2 continues to be displayed. At that time, the screen background is changed to the display 109T indicating the transition area.

[0125] (S3'): Position F2 of finger PS is within the range of layer L1, and icon group 2102 corresponding to applications E to H is displayed on the window screen in a switching manner. (S2'): When the position F1 of the finger PS moves to the layer transition area T01 between the layers L0 and L1, the window screen (icon group 2102) displayed on the previous layer L1 continues to be displayed. At that time, the screen background is changed to the display 109T indicating the transition area.

[0126] (S1'): The position F0 of the finger PS is within the range of the layer L0, and the icon group 2101 corresponding to the applications A to D is displayed on the window screen in a switching manner.

[0127] As described above, when the finger PS is located within the range of each layer, a window screen corresponding to that layer is displayed, and when the finger PS moves to the layer transition area, the window screen of the layer in which the finger PS was located immediately before continues to be displayed, and the background is changed and displayed.

[0128] In this way, a transition area is provided between each layer, and when the finger PS moves into the transition area, the background of the window screen changes to notify the user that they have moved into the transition area. In this case, the previously displayed window screen continues as is, so there is no disruption to operation. This prevents the user from unintentionally moving to another layer. In other words, when the background of the window screen changes, all they have to do is move the finger PS in the direction of the original layer.

[0129] 24 is a flowchart showing a mouse operation input process when a transition area is provided between layers. The same processes as those in the third embodiment (FIG. 17) and the fourth embodiment (FIG. 21) are denoted by the same reference numerals, and repeated explanations will be omitted.

[0130] In S1501, the three-dimensional position F of the finger PS is acquired, and in S1502, the position of position F in the X and Y directions is converted to position f in the display range Da. Next, in S1520, it is determined which layer range the position of position F in the Z direction falls within, or whether it is within a layer transition area. In S1530, branching processing is performed depending on the layer determination result.

[0131] In the determination process S1530, if the determined layer LN is within the layer transition area (LN=between layers), the process branches to S1531. In S1531, the background of the window display screen is set to the display 109T corresponding to the layer transition area, and the process proceeds to S1503.

[0132] In the determination process S1530, if the determined layer LN is the same as the previously determined layer LNd (LN=LNd), the process proceeds to S1503. The process from S1503 onwards is the same as in Fig. 17, and the description will be omitted.

[0133] In the determination process S1530, if the determined layer LN is different from the previously determined layer LNd (LN≠LNd), the process proceeds to S1522. The process from S1522 onwards is the same as in Fig. 21, and the description will be omitted.

[0134] As described above, according to the fifth embodiment, the three-dimensional position of the finger PS is detected, and windows corresponding to multiple layers are displayed according to the position in the height direction Z. In addition, transition areas are provided between each layer. When the finger PS moves into the transition area, the background of the window screen is changed to notify the user, thereby preventing the user from unintentionally moving to another layer. This allows smooth switching between each layer.

[0135] In this embodiment, the background of the window screen is changed when the finger PS moves into the layer transition area, but this is not limiting. For example, any method that can be recognized by the user may be used, such as changing the icon display, changing the cursor display, or generating a sound or vibration. Furthermore, when the finger PS is positioned in the transition area, the direction of the original layer (up / down) may be displayed.

[0136] In the above description of each embodiment, an example was shown in which the finger position detection sensor 106 is configured with a light emitting / receiving element, but the present invention is not limited to this and any other sensor capable of detecting the position of a position input object such as a user's finger or pen, such as an ultrasonic sensor or an imaging element such as a camera, may be used. Also, while a wristwatch-type input terminal device has been described as an example, a smartphone, tablet, personal computer, etc. may also be configured to detect the position of a position input object such as a user's finger or pen and perform handwritten character input, mouse operation, etc.

[0137] In the above embodiments, the condition for a specific operation for switching the operation mode is that the position of a position input object, such as a user's finger, on the display screen is held within a predetermined distance for a predetermined period of time or more. However, this is not limiting. For example, the condition may be that the amount of movement of the position of the position input object on the display screen within a predetermined period of time is within a predetermined distance. Alternatively, the condition may be that the speed at which the position of the position input object on the display screen is moving is equal to or less than a predetermined speed.

[0138] Although the embodiments of the present invention have been described above using several examples, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above examples, and various modifications are possible. For example, it is possible to replace part of the configuration of one example with the configuration of another example, or it is also possible to add the configuration of one example to the configuration of another example. All of these fall within the scope of the present invention. Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention are not impaired even if different ones are used. Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. Furthermore, the order in which each process is performed may be changed.

[0139] The functions of the present invention described above may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Alternatively, they may be realized by software, such as by a microprocessor unit interpreting and executing an operating program that realizes each function. Hardware and software may also be used in combination.

[0140] Furthermore, the control lines and information lines shown in the drawings are those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0141] 1: input terminal device, 2: mobile information terminal, 101: main control unit, 102: ROM, 103: RAM, 104: storage unit, 105: acceleration sensor, 106, 106a, 106b, 106c: finger position detection sensors, 108: display control unit, 109: touch panel, 109a: display unit, 109b: touch operation input unit, 110: timing unit, 111: operation input unit, 1022: finger position information acquisition unit, 1023: touch position information acquisition unit, 1024: finger operation processing control unit, 1025: touch operation processing control unit, PS: user's finger (position input object).

Claims

1. A terminal device capable of inputting a user's operation via a position input object, a position detection unit that detects the three-dimensional position of the position input object in a non-contact manner; a display unit that displays an operation target; an operation processing control unit that executes a corresponding operation process based on the three-dimensional position detected by the position detection unit; The operation processing control unit when the position detection unit detects that the position input object is continuously at a three-dimensional position corresponding to the operation object displayed on the display unit for a predetermined time or more, control is performed to switch from a first operation mode in which operation processing is not performed on the operation object to a second operation mode in which operation processing is performed on the operation object; Furthermore, a vertical position of the position input object relative to the display unit among the three-dimensional positions of the position input object relative to the display unit is divided into a plurality of layers, and the display content of the display unit is changed depending on which of the plurality of layers the vertical position of the position input object detected by the position detection unit falls within; A terminal device characterized in that a transition area is further set at the boundary between the plurality of layers, and when the position of the position input object moves into the transition area, the display unit continues to display the operation screen that was displayed immediately before, while notifying the user that the position of the position input object is within the transition area.

2. 2. The terminal device according to claim 1, The terminal device is characterized in that the operation control processing unit controls the display unit to change the display of the operation object when the vertical position of the position input object relative to the display unit is within a predetermined first range in the second operation mode.

3. 2. The terminal device according to claim 1, The operation control processing unit divides the vertical position relative to the display unit into multiple layers, and controls the display content of the display unit to change depending on which of the multiple layers the vertical position of the position input object detected by the position detection unit falls within.

4. The terminal device according to claim 3, The operation processing control unit further sets a transition area at the boundary between the plurality of layers, and when the position of the position input object moves into the transition area, the display unit continues to display the operation screen that was displayed immediately before, while changing the background of the screen.

5. 5. A terminal device according to claim 1, The terminal device is characterized in that the position detection unit is configured to include a light emitting element and a light receiving element.

6. 5. A terminal device according to claim 1, The terminal device is characterized in that the position detection unit is configured to include a sensor that uses ultrasonic waves.

7. 5. A terminal device according to claim 1, The terminal device is characterized in that the position detection unit is configured to include a camera.

8. 2. The terminal device according to claim 1, Furthermore, it is equipped with a touch panel that detects touch operations, A terminal device characterized by having a first operation input mode that executes a corresponding operation process based on a touch operation on the touch panel, and a second operation input mode that executes a corresponding operation process based on a three-dimensional position detected by the position detection unit.

9. 9. The terminal device according to claim 8, The terminal device further comprises an operation input unit that is switchable from the first operation input mode to the second operation input mode.

10. 9. The terminal device according to claim 8, When a predetermined operation by the position input object is detected, the terminal device switches to the first operation input mode or the second operation input mode.

Images