Operation input system

Abstract

To provide an operation input system allowing an operator to perform input operation accurately by a finger.SOLUTION: An operation input system according to the present invention includes: a first sensor device which has an annular base body being wearable to a first finger of an operator, and outputs a first signal according to a motion of the first finger; a second sensor device which has an annular base body being wearable to a second finger different from the first finger and outputs a second signal according to a motion of the second finger; and a control device which determines a relative finger motion of the first finger and the second finger on the basis of the first signal and the second signal.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operation input system. [Background technology]

[0002] Patent Document 1 discloses a smart interface ring that has a pressure-sensitive touch pointer and a three-axis gyro sensor and is worn on one of the operator's fingers. This smart interface ring captures the movement of the fingertip based on the angular velocity of the finger detected by the three-axis gyro sensor, enabling operation input for an application. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-21270 A Summary of the Invention [Problem to be solved by the invention]

[0004] The ring described in Patent Document 1 is configured to detect only the angular velocity of one finger of a hand in a specified posture to determine its movement, so there is an issue that it cannot accurately determine the movement of the operator's finger, resulting in low accuracy of input operations.

[0005] In view of the above-mentioned problems in the conventional technology, an object of the present invention is to provide an operation input system that allows an operator to perform input operations with finger movements with high accuracy. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an operation input system including a first sensor device having a ring-shaped base that can be worn on a first finger of an operator and that outputs a first signal corresponding to a movement of the first finger, a second sensor device having a ring-shaped base that can be worn on a second finger different from the first finger and that outputs a second signal corresponding to a movement of the second finger, and a control device that determines a relative finger movement of the first finger and the second finger based on the first signal and the second signal.

[0007] According to another aspect of the present invention, there is provided an operation input system including a first sensor device that is worn on a first finger of an operator and detects a first signal corresponding to a movement of the first finger, a second sensor device that is worn on a second finger different from the first finger and detects a second signal corresponding to a movement of the second finger, and a control device that controls the first sensor device and the second sensor device, respectively, wherein the control device detects a relative finger movement of the first finger and the second finger based on the first signal and the second signal, and, on a condition that it has determined that the finger movement is a predetermined first gesture, determines whether the finger movement performed following the first gesture is a predetermined second gesture, and when it has determined that the finger movement is the second gesture, outputs a command corresponding to the second gesture. Effect of the Invention

[0008] According to the present invention, there is provided an operation input system that enables an operator to perform an input operation with a finger movement with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of devices constituting an operation input system according to a first embodiment. [Diagram 2] 3A to 3C are diagrams illustrating an example of how a first sensor device and a second sensor device according to the first embodiment are attached to an operator's hand. [Diagram 3] 3 is a cross-sectional view showing an example of an internal structure of a first sensor device according to the first embodiment. FIG. [Figure 4]3 is a cross-sectional view showing an example of an internal structure of a first sensor device according to the first embodiment. FIG. [Diagram 5] FIG. 2 is a functional block diagram showing an example of functions of a control device according to the first embodiment. [Figure 6] 5A to 5C are diagrams illustrating a change over time in the angular velocity of the thumb detected by the first sensor device according to the first embodiment and an extraction period for the peak value. [Figure 7] 5 is a diagram showing a change over time in angular velocity of the thumb detected by the first sensor device according to the first embodiment. FIG. [Figure 8] 6 is a diagram showing a change over time in the angular velocity of the index finger detected by the second sensor device in the first embodiment. FIG. [Figure 9] 4 is a timing chart showing processes and execution times executed in the control device according to the first embodiment. [Figure 10] 4 is a flowchart showing an example of processing executed in the operation input system according to the first embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of an overall configuration of an operation input system according to a second embodiment. [Figure 12] FIG. 11 is a functional block diagram showing an example of functions of a control device according to a second embodiment. [Figure 13] 13A to 13C are diagrams illustrating the relationship between finger motions and finger state transitions in the operation input system according to the second embodiment. [Figure 14] 13 is a diagram showing a combination of peak values ​​of angular velocity of the thumb used in determining each gesture in the operation input system according to the second embodiment. FIG. [Figure 15] 13 is a diagram showing a combination of peak values ​​of angular velocity of the index finger used in determining each gesture in the operation input system according to the second embodiment. FIG. [Figure 16] 13 is a top view illustrating a plurality of gestures detected in the operation input system according to the second embodiment. FIG. [Figure 17] FIG. 17 is a side view illustrating the gesture shown in FIG. [Figure 18] FIG. 17 is a side view illustrating the gesture shown in FIG. [Figure 19] 10 is a flowchart showing an example of processing executed in the operation input system according to the second embodiment. [Figure 20] FIG. 11 is a diagram showing an example of a screen displayed on a control device according to the second embodiment. [Figure 21] FIG. 11 is a block diagram showing an example of a hardware configuration of devices constituting an operation input system according to a third embodiment. [Figure 22] FIG. 11 is a perspective view showing an example of smart glasses according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Elements having common functions are designated by the same reference numerals throughout the drawings, and duplicated descriptions may be omitted or simplified.

[0011] [First embodiment] FIG. 1 is a block diagram showing an example of a hardware configuration of devices constituting an operation input system 1 according to this embodiment.

[0012] 1, the operation input system 1 includes a first sensor device 10, a second sensor device 20, and a control device 30. In this embodiment, the first sensor device 10 and the second sensor device 20 are connected to the control device 30 via a network.

[0013] The first sensor device 10 has a ring-shaped base body that can be worn on a first finger of an operator, and outputs a first signal corresponding to the movement of the first finger. The first sensor device 10 of this embodiment outputs detection data of the angular velocity of the first finger (hereinafter referred to as "first angular velocity") as a first signal corresponding to the movement of the first finger. Specifically, the first sensor device 10 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

[0014] 1, the first sensor device 10 includes an MCU (Micro Controller Unit) 101, a sensor module 102, a feedback module 103, a wireless communication device 104, and a battery 105. The components of the first sensor device 10 are connected by wiring on a circuit board (not shown).

[0015] The MCU 101 is an embedded microprocessor that integrates a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), I / O devices, etc. into a single integrated circuit. The MCU 101 is primarily used to control electronic devices.

[0016] The sensor module 102 has a gyro sensor that detects angular velocity. The gyro sensor of this embodiment is a three-axis gyro sensor that detects angular velocity around the mutually orthogonal X-axis, Y-axis, and Z-axis. In addition to the gyro sensor, the sensor module 102 may further include sensors such as an acceleration sensor and a magnetic sensor. When an acceleration sensor is used, the actual acceleration is calculated by removing the gravity component from the detected acceleration value.

[0017] The sensor module 102 applies a voltage proportional to the detected angular velocity to the MCU 101. The MCU 101 performs A / D conversion of the voltage applied from the sensor module 102 into digital data. Then, the MCU 101 transmits the detected data of the angular velocity to the control device 30 via a wireless communication device 104, which will be described later.

[0018] The feedback module 103 notifies the operator who wears the first sensor device 10 on his or her thumb, based on the control information received from the control device 30. The feedback module 103 will be described in detail later.

[0019] The wireless communication device 104 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for performing wireless communication with other devices. The wireless communication device 104 transmits detection data of the first angular velocity to the control device 30 by wireless communication.

[0020] The battery 105 is a device that supplies power for driving each of the MCU 101, the sensor module 102, the feedback module 103, and the wireless communication device 104.

[0021] The second sensor device 20 has a ring-shaped base that can be attached to a second finger different from the first finger of the operator, and outputs a second signal corresponding to the movement of the second finger. The second sensor device 20 of this embodiment outputs detection data of the angular velocity of the second finger (hereinafter referred to as the "second angular velocity") as a second signal corresponding to the movement of the second finger. Specifically, the second sensor device 20 transmits the detection data of the second angular velocity to the control device 30 by wireless communication.

[0022] In this embodiment, the first finger is the thumb. The second finger is preferably the finger adjacent to the first finger, and is therefore the index finger. The setting information in the first sensor device 10 and the second sensor device 20 is set according to the thumb and index finger of the wearer.

[0023] 1, the second sensor device 20 includes an MCU 201, a sensor module 202, a feedback module 203, a wireless communication device 204, and a battery 205. The hardware configuration of the second sensor device 20 is similar to that of the first sensor device 10, and therefore a description of each device will be omitted.

[0024] The control device 30 is a computer device that performs calculations, control, and storage. Examples of the control device 30 include a personal computer, a laptop computer, a tablet terminal, a smartphone, etc. The control device 30 determines the relative finger movements of the first finger and the second finger based on the first angular velocity and the second angular velocity.

[0025] 1, the control device 30 includes a processor 301, a RAM 302, a ROM 303, a storage 304, a communication I / F (Interface) 305, a display device 306, and an input device 307. Each device is connected to each other via a bus, wiring, a drive device, etc.

[0026] The processor 301 has a function of performing predetermined calculations according to programs stored in the ROM 303, the storage 304, etc., and controlling each part of the control device 30, the first sensor device 10, and the second sensor device 20. As the processor 301, a CPU, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), etc. are used.

[0027] The RAM 302 is a storage device that is configured from a volatile storage medium and provides a temporary memory area necessary for the operation of the processor 301. The RAM 302 may be, for example, a D-RAM (Dynamic RAM).

[0028] The ROM 303 is configured from a non-volatile storage medium, and stores necessary information such as programs used in the operation of the control device 30. The ROM 303 may be, for example, a P-ROM (Programmable ROM).

[0029] The storage 304 is a storage device that is configured from a non-volatile storage medium and stores data processed by the control device 30 and programs for operating the control device 30. The storage 304 is configured from, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0030] The processor 301 loads a program stored in the ROM 303, the storage 304, etc., into the RAM 302 and executes it.

[0031] The communication I / F 305 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G or 5G, and is a module for performing wireless communication with the first sensor device 10 and the second sensor device 20.

[0032] The display device 306 displays moving images, still images, characters, etc. A liquid crystal display, an OLED (Organic Light Emitting Diode) display, etc. are used as the display device 306. The operation input screen displayed on the display device 306 includes graphics such as windows, icons, buttons, etc. The operator selects a graphic representing a desired operation on the operation input screen by operation input means using a pointing device, or operation input means using the first sensor device 10 and the second sensor device 20, etc.

[0033] The input device 307 is a keyboard, a pointing device, etc., and is used by an operator to operate the control device 30. Examples of the pointing device include a mouse, a trackball, a touch panel, a pen tablet, etc. The display device 306 and the input device 307 may be integrally formed as a touch panel.

[0034] The hardware configuration shown in FIG. 1 is an example, and other devices may be added, or some devices may not be provided. Some devices may be replaced with other devices having similar functions. Some functions of the first embodiment may be provided by other devices via a network, or the functions of the first embodiment may be distributed and realized among a plurality of devices. The illustrated hardware configuration can be changed as appropriate.

[0035] Fig. 2 is a diagram showing an example of how the first sensor device 10 and the second sensor device 20 according to this embodiment are attached to the fingers of an operator. In Fig. 2, the first sensor device 10 is attached to the base side of the proximal phalanx of the operator's thumb F1. The second sensor device 20 is attached to the base side of the proximal phalanx of the operator's index finger F2.

[0036] In the XYZ coordinate system of the second sensor device 20, the X axis is set in the width direction of the operator's index finger F2, the Y axis is set in the direction in which the proximal phalanx of the index finger F2 extends, and the Z axis is set in the thickness direction of the index finger F2. The X axis, Y axis, and Z axis are perpendicular to each other. Note that, similarly to the second sensor device 20, in the XYZ coordinate system of the first sensor device 10, the X axis is set in the width direction of the operator's thumb F1, the Y axis is set in the direction in which the proximal phalanx of the thumb F1 extends, and the Z axis is set in the thickness direction of the thumb F1.

[0037] 3 and 4 are cross-sectional views showing an example of the internal structure of the first sensor device 10 according to this embodiment. Note that the internal structure of the second sensor device 20 is similar to that of the first sensor device 10, so the first sensor device 10 will be described below as a representative example.

[0038] FIG. 3 shows an example in which a tightening mechanism 103A is used as the feedback module 103 in the first sensor device 10. The tightening mechanism 103A expands or contracts based on control information from the control device 30. Examples of the tightening mechanism 103A include an airbag or a dielectric member (e.g., e-Rubber, a next-generation rubber material manufactured by Toyoda Gosei Co., Ltd.) that can expand or contract depending on an applied voltage. When the tightening mechanism 103A expands, it deforms into a protrusion toward the operator's finger, and a pressing force is applied to the operator's finger. As a result, the control device 30 causes the feedback module 103 to execute feedback on the operation input from the operator. The feedback is performed to notify the operator that the input operation by the operator has been accepted and a predetermined command has been executed.

[0039] 3, the tightening mechanism 103A is formed in a ring shape like the base 11, but the shape of the tightening mechanism 103A is not limited to this. The tightening mechanism 103A may be formed in a sheet shape, for example. Also, the number of the tightening mechanism 103A is one, but is not limited to this.

[0040] When multiple tightening mechanisms 103A are provided, there are multiple locations within the device that can expand or contract. Therefore, feedback in a clockwise or counterclockwise direction, etc., can be achieved by switching the tightening mechanism 103A to be expanded or contracted in the first sensor device 10 according to time. In other words, it is possible to further increase the variety of feedback methods.

[0041] The first sensor device 10 also includes three sensor modules 102. The three sensor modules 102 are arranged at approximately equal intervals around the circumference of the base body 11 in a cross-sectional view. A sensor such as a gyro sensor detects different changes in the value of angular velocity depending on the mounting position. Therefore, by providing the three sensor modules 102 at a distance from each other, it is possible to detect angular velocity at different positions. As a result, the control device 30 can determine the mounting state of the first sensor device 10 on the thumb F1 with high accuracy based on the multiple angular velocities. The same is true for the second sensor device 20.

[0042] 4 shows an example in which three vibration mechanisms 103B are used as the feedback module 103 in the first sensor device 10. The number of sensor modules 102 is three, similar to the case of FIG.

[0043] The vibration mechanism 103B vibrates based on control information from the control device 30. The vibration mechanism 103B is composed of, for example, a piezoelectric element (piezo element) that can expand or contract according to an applied voltage, and an electromagnetic drive unit such as a vibration motor that can vibrate. A piezoelectric element is a passive element that performs micro-movement control and detection by utilizing the piezoelectric effect and inverse piezoelectric effect that occur in dielectric materials such as crystal or quartz. Since it has a simple structure that does not require gears or motors for operation, it is a smaller element than other micro-movement mechanism elements.

[0044] Furthermore, the vibration mechanism 103B does not have to be provided adjacent to the sensor module 102, and may be provided between a plurality of sensor modules 102. When the vibration mechanism 103B vibrates, the vibration is transmitted to the operator's finger. As a result, the control device 30 causes the feedback module 103 to execute feedback for the operation input from the operator.

[0045] 4, the number of vibration mechanisms 103B is the same as the number of sensor modules 102, but the number of vibration mechanisms 103B is not limited to this. The magnitude and period of vibration in each module may be changed.

[0046] When multiple vibration mechanisms 103B are provided, there are multiple locations in the device that can vibrate. Feedback in a clockwise or counterclockwise direction, etc., can be achieved by switching the vibration mechanism 103B that vibrates in the first sensor device 10 according to time. In other words, it is possible to further increase the variety of feedback methods.

[0047] 5 is a functional block diagram showing an example of functions of the control device 30 according to this embodiment. The control device 30 includes a calibration unit 30A, an angular velocity calculation unit 30B, a peak value extraction unit 30C, a command issuing unit 30D, a feedback control unit 30E, a display control unit 30F, and an input unit 30G.

[0048] When the first sensor device 10 and the second sensor device 20 are attached to an operator's finger and activated for the first time, the calibration unit 30A performs calibration on the detection data of the angular velocity detected by the sensor modules 102, 202. The calibration unit 30A outputs a correction value (offset value) of the angular velocity calculated by the calibration to the angular velocity calculation unit 30B.

[0049] The angular velocity calculation unit 30B calculates the angular velocities of the thumb F1 and index finger F2 of the operator based on the detection data of the angular velocities acquired from the first sensor device 10 and the second sensor device 20 by wireless communication and the correction value calculated by the calibration unit 30A. The angular velocity calculation unit 30B outputs the calculated angular velocities to the peak value extraction unit 30C. The calculation process of the angular velocity in the angular velocity calculation unit 30B is repeatedly executed every time the detection data of the angular velocity is received from the first sensor device 10 and the second sensor device 20.

[0050] Peak value extractor 30C extracts a peak value from the angular velocity within a predetermined extraction period, and outputs the extracted peak value to command issuer 30D.

[0051] FIG. 6 is a diagram showing an example of the time change of the angular velocity detected by the first sensor device 10 according to the present embodiment and the extraction period of the peak value. In FIG. 6, as shown by the dashed line, the positive threshold value THp of the angular velocity is set to +20 rad / sec, and the negative threshold value THm is set to -20 rad / sec. The absolute values ​​of the threshold values ​​THp and THm are equal. The extraction period T is set to a time t2 after a certain time has elapsed from the time t1 when at least one of the angular velocities around the X-axis, Y-axis, and Z-axis of two fingers becomes equal to or greater than a predetermined threshold value. However, the method of setting the values ​​of the threshold values ​​THp and THm and the extraction period T is not limited to this.

[0052] The command issuing unit 30D determines the relative finger motion of the operator's two fingers based on the peak value extracted within the extraction period, and outputs a command that is pre-associated with the finger motion. The command issuing unit 30D detects at least two types of finger motion. When the command issuing unit 30D outputs a command, a process corresponding to the command is executed in one of the devices that configure the operation input system 1.

[0053] The feedback control unit 30E transmits control information to the MCU 101 of the first sensor device 10 and the MCU 201 of the second sensor device 20, and causes the feedback modules 103, 203 to execute predetermined feedback processing.

[0054] The display control unit 30F controls the display of an operation input screen displayed on the display device 306 based on an input operation by the input device 307 received by the input unit 30G and an input operation by the first sensor device 10 and the second sensor device 20.

[0055] The input unit 30G receives information input by a user such as an operator or a manager using the input device 307 to operate the control device 30.

[0056] Fig. 7 is a diagram showing a change over time in the angular velocity of the thumb F1 detected by the first sensor device 10 according to this embodiment. Meanwhile, Fig. 8 is a diagram showing a change over time in the angular velocity of the index finger F2 detected by the second sensor device 20 according to this embodiment. Figs. 7 and 8 show an extraction period A from times t1 to t2, an extraction period B from times t3 to t4, and an extraction period C from times t5 to t6.

[0057] 7 are different from the thresholds THp and THm for the angular velocity of the index finger F2 in FIG. 8. In this manner, it is preferable to set the thresholds THp and THm to optimal values ​​for each finger. The control device 30 determines the relative finger motion of the thumb F1 and index finger F2 in each of the extraction periods A, B, and C based on the peak values ​​of the angular velocities of the two fingers around the X-axis, Y-axis, and Z-axis and the thresholds THp and THm in the extraction periods A, B, and C.

[0058] 7, in the extraction period A, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is greater than the threshold value THp on the positive side. The angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold value THm on the negative side. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp.

[0059] 8, in the extraction period A, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold value THp. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the positive side, and the peak value on the positive side is greater than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on both the positive and negative sides, and the peak value on the positive side is smaller than the threshold value THp and the peak value on the negative side is greater than the threshold value THm.

[0060] 7 and 8, it can be seen that the thumb F1 moves significantly in the +X direction around the X axis during extraction period A. It can also be seen that the index finger F2 moves significantly in the +Y direction around the Y axis during extraction period A. As a result, a finger motion in which the thumb F1 is raised above the index finger F2 during extraction period A is detected.

[0061] 7, in the extraction period B, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0062] 8, in the extraction period B, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp.

[0063] 7 and 8, it can be seen that the thumb F1 moves significantly in the -X direction around the X axis during extraction period B. It can also be seen that the index finger F2 moves significantly in the -Y direction around the Y axis during extraction period B. As a result, a finger motion in which the thumb F1 touches the index finger F2 is detected during extraction period B.

[0064] 7, in the extraction period C, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0065] 8, in the extraction period C, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0066] 7 and 8 for extraction period C, it can be seen that the thumb F1 moves significantly in the +Y direction around the Y axis. It can also be seen that the index finger F2 moves significantly in the -X direction around the X axis and in the +Y direction around the Y axis during extraction period C. As a result, a finger motion is detected in which the thumb F1 keeps contact with the index finger F2 and slides to the right along the side of the index finger F2 during extraction period C.

[0067] Fig. 9 is a timing chart showing the processes and execution times executed by the control device 30 according to this embodiment. Here, the correspondence between the extraction periods A, B, and C shown in Fig. 7 and Fig. 8 and the command issuing process, the feedback destination determination process, the drive control process for the feedback module, and the display control process for the operation input screen is shown.

[0068] The first command issuing process in the command issuing unit 30D is executed during extraction period A from time t1 to t2. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t2 to t21. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t21 to t22.

[0069] Furthermore, the second command issuing process in the command issuing unit 30D is executed during extraction period B from time t3 to t4. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t4 to t41. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t41 to t42.

[0070] The third command issuing process in the command issuing unit 30D is executed during extraction period C from time t5 to t6. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t6 to t61. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t61 to t62.

[0071] 10 is a flowchart showing an example of a process executed by the control device 30 according to the present embodiment. This process is executed repeatedly while the operation input screen is displayed, for example.

[0072] First, the control device 30 acquires detection data of angular velocity from the first sensor device 10 and the second sensor device 20 by wireless communication (step S101).

[0073] Next, the control device 30 determines whether or not the first sensor device 10 and the second sensor device 20 are being started for the first time (step S102). If the control device 30 determines that the first sensor device 10 and the second sensor device 20 are being started for the first time (step S102: YES), the process proceeds to step S103. On the other hand, if the control device 30 determines that the first sensor device 10 and the second sensor device 20 are not being started for the first time (step S102: NO), the process proceeds to step S105.

[0074] In step S103, the control device 30 executes calibration based on the detection data of the angular velocity acquired from the first sensor device 10 and the second sensor device 20, and calculates a correction value of the angular velocity. After that, the process proceeds to step S104.

[0075] In step S104, the control device 30 determines the mounting state of the sensor based on the detection data of the angular velocity. After that, the process proceeds to step S105.

[0076] 3, the first sensor device 10 includes a plurality of sensor modules 102. Therefore, the control device 30 can determine the attachment state of the first sensor device 10 on the thumb F1 based on a plurality of detection data detected by the plurality of sensor modules 102.

[0077] Similarly, the second sensor device 20 includes a plurality of sensor modules 202. The control device 30 can determine the attachment state of the second sensor device 20 on the index finger F2 based on a plurality of detection data detected by the plurality of sensor modules 202.

[0078] In step S105, the control device 30 calculates the angular velocities of the thumb F1 and the index finger F2 based on the detection data of the angular velocities and the correction value of the angular velocities. The angular velocities of the thumb F1 and the index finger F2 are calculated for three axes, the X-axis, the Y-axis, and the Z-axis.

[0079] Next, the control device 30 determines whether or not any of the absolute values ​​of the calculated angular velocities is equal to or greater than a predetermined threshold value (step S106).

[0080] If the control device 30 determines that the absolute value of any one of the angular velocities is equal to or greater than the predetermined threshold (step S106: YES), the process proceeds to step S107. On the other hand, if the control device 30 determines that the absolute values ​​of all of the angular velocities are less than the predetermined threshold (step S107: NO), the process returns to step S101.

[0081] In step S107, the control device 30 determines an extraction period for the angular velocity, and extracts peak values ​​from the angular velocity within the extraction period. Peak values ​​are extracted for each of the three axes, the X-axis, the Y-axis, and the Z-axis. If the angular velocity fluctuates within the extraction period, multiple peak values ​​may be extracted in the same direction. In such a case, the control device 30, for example, successively updates the peak values ​​stored in the storage area.

[0082] Next, the control device 30 determines the relative finger motion of the two fingers based on a combination of the peak values ​​of the angular velocities of the two fingers extracted within the extraction period (step S108). Since the finger motion determination is started when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 exceeds a predetermined threshold, the calculation load is suppressed.

[0083] In step S108, for example, finger movements such as moving the thumb F1 and index finger F2 apart, bringing the thumb F1 and index finger F2 into contact, and finger movements in which the thumb F1 moves up and down, left and right, or back and forth along the index finger F2 while the thumb F1 and index finger F2 are in contact are determined.

[0084] Next, when the control device 30 determines a command corresponding to the relative motion of the two fingers, it outputs the command (step S109). As a result, the control device 30 executes the process specified by the command. Examples of the command include a command to move the position of the cursor or pointer on the operation input screen up, down, left, or right, a command to cancel the operation input performed immediately before, and a command to confirm the operation input.

[0085] Next, the control device 30 determines a feedback destination in accordance with the execution of the issued command (step S110). In this embodiment, the first sensor device 10, the second sensor device 20, and the display device 106 can be feedback destinations.

[0086] Next, the control device 30 transmits control information to the feedback modules 103, 203 that are the determined feedback destinations, and drives the feedback modules 103, 203 (step S111).

[0087] Next, the control device 30 updates the operation input screen displayed on the display device 106 (step S112). For example, the control device 30 changes the coordinates of the pointer on the operation input screen, or changes the display state of the selected menu. In this way, the control device 30 notifies the GUI (display device) in synchronization with the notification to the feedback module. Then, the process ends.

[0088] Then, the control device 30 judges whether or not the end of the operation input has been instructed by the operator on the operation input screen (step S113). If the control device 30 judges that the end of the operation input has been instructed (step S113: YES), the process ends. On the other hand, if the control device 30 judges that the end of the operation input has not been instructed (step S113: NO), the process returns to step S101.

[0089] Conventionally, input means using gestures or controllers, input means using eye gaze, input means using voice, etc. are known as input means that can replace mouse operation. However, input means using gestures or controllers require large operation input movements, which makes the operation state visible to others. In addition, with input means using eye gaze, the eye gaze position fluctuates, making stable operation difficult. And with input means using voice, the voice giving instructions to operate can be heard by others. In view of these problems, there has been a demand for a pointing device that is difficult for others to recognize and has excellent operability.

[0090] In contrast, according to the operation input system 1 of this embodiment, a first sensor device 10 and a second sensor device 20, which are ring-shaped devices equipped with a gyro sensor, are attached to the thumb (first finger) F1 and the index finger (second finger) F2, respectively. The first sensor device 10 and the second sensor device 20 transmit the angular velocities of the two fingers to the control device 30 by wireless communication. The control device 30 detects the relative finger motion of the two fingers using the angular velocities of the thumb F1 and the index finger F2.

[0091] This allows the system to determine and output commands corresponding to finger movements, allowing users to freely and accurately input operations. In addition, because operations can be input simply by moving the thumb F1 and index finger F2, it is difficult for others to know that the operator is inputting operations, which is beneficial from the standpoint of improving information security and protecting privacy.

[0092] In addition, in conventional systems where a ring-shaped device was attached to only one finger, the direction of the arm had to be changed in order to point to a specific position with a finger. In contrast, the operation input system 1 according to this embodiment is configured to determine the relative motion of two fingers by attaching a ring-shaped device equipped with a gyro sensor to each of the thumb F1 and index finger F2. This makes it possible to point with any arm direction.

[0093] Also, among conventional ring-shaped devices, those equipped with a gyro sensor for detecting motion are known. However, since the information that can be detected varies depending on the mounting position of the gyro sensor, the ring-shaped device must always be worn in a fixed direction. In contrast, according to the operation input system 1 of this embodiment, the ring-shaped device (the first sensor device 10 and the second sensor device 20) is equipped with multiple sensor modules at different positions within the ring. Since the mounting state of the ring-shaped device can be determined based on multiple detection data from multiple sensor modules, the ring-shaped device can be worn in any position on each finger.

[0094] [Second embodiment] The operation input system 2 according to the second embodiment will be described below. The following mainly describes the differences from the first embodiment, and the description of the common parts will be omitted or simplified.

[0095] 11 is a block diagram showing an example of a hardware configuration of the operation input system 2 according to this embodiment. The hardware configuration of the operation input system 2 is common to the hardware configuration of the operation input system 1 according to the first embodiment.

[0096] 12 is a functional block diagram showing an example of functions of the control device 30 according to this embodiment. Unlike the control device 30 in the first embodiment, the control device 30 according to this embodiment further includes a gesture detection unit 30H.

[0097] The gesture detection unit 30H detects a specific finger motion from among arbitrary finger motions by the thumb F1 and index finger F2 based on the angular velocity of the thumb F1 and index finger F2. In this embodiment, a specific finger motion that the gesture detection unit 30H detects from among arbitrary finger motions is called a "gesture". Note that a gesture is not limited to a single finger motion. There may be cases where multiple finger motions performed consecutively within a certain period of time are defined as one gesture. It is assumed that the correspondence between multiple gestures and commands issued for each gesture is defined in advance. The command issuing unit 30D of this embodiment outputs a command corresponding to a specific gesture detected by the gesture detection unit 30H.

[0098] 13 is a diagram showing the relationship between finger motions and finger state transitions in the operation input system 2 according to this embodiment. Here, "thumb up", "thumb down", "thumb down", "stationary", and "moving" are shown as finger motions. Also, "initial (ST0)", "left (ST1)", "right (ST2)", "forward (ST3)", "backward (ST4)", "down (ST5)", "released (ST6)", and "contact (ST7)" are shown as finger states.

[0099] "Initial (ST0)" is a state in which the operator lightly closes his / her hand and places the thumb F1 at the midpoint between the first and second joints of the index finger F2. Hereinafter, "initial (ST0)" will be referred to as the initial state ST0.

[0100] "Left (ST1)" is a state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved leftward along the side of the index finger F2. Hereinafter, "left (ST1)" will be referred to as state ST1.

[0101] "Right (ST2)" is a state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved to the right along the side of the index finger F2. Hereinafter, "right (ST2)" is referred to as state ST2.

[0102] "Front (ST3)" is a state in which the thumb F1 and index finger F2 in the initial state ST0 are in contact with each other, and the tip of the thumb F1 is pushed outward from the base of the index finger F2. Hereinafter, "front (ST3)" will be referred to as state ST3.

[0103] "Back (ST4)" is a state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved to the side of the base of the index finger F2. Hereinafter, "back (ST4)" is referred to as state ST4.

[0104] "Down (ST5)" is a state in which the thumb F1 and index finger F2 in the initial state ST0 are in contact with each other, and the tip of the thumb F1 is pressed down to a position below the index finger F2. Hereinafter, "down (ST5)" will be referred to as state ST5. In state ST5, the thumb F1 is located between the index finger F2 and the middle finger F3.

[0105] "Release (ST6)" is a state in which the thumb F1 and index finger F2, which have been in contact with each other, are separated and maintained in this state for a certain period of time. Hereinafter, "release (ST6)" will be referred to as the release state ST6.

[0106] "Contact (ST7)" is a state in which the thumb F1 and index finger F2, which were in the released state ST6, are in contact with each other. Hereinafter, "Contact (ST7)" will be referred to as state ST7. State ST7 is an intermediate state when changing from the released state ST6 to the initial state ST0.

[0107] The control device 30 (gesture detection unit 30H) determines the state of the thumb F1 and index finger F2 based on the change in angular velocity. When the state transitions from the release state ST6 to the contact state ST7 and the contact state ST7 continues for a certain period of time, the control device 30 determines that the current state is the initial state ST0.

[0108] In addition, in this embodiment, the first gesture and the second gesture are defined as follows. (1) First gesture The first gesture is a motion of moving the thumb F1 and index finger F2 from any of states ST0 to ST5 to the initial state ST0 via states ST6 and ST7. In other words, the first gesture is a motion of lifting the thumb F1 and lowering the thumb F2. (2) Second Gesture The second gesture includes the following two types of actions (A) and (B). (A) An action of putting the thumb F1 and the index finger F2 into one of states ST1 to ST5, starting from the initial state ST0. (B) A series of actions in which a similar action (ST0 → ST6 → ST7 → ST0) is performed within a certain period of time immediately following the first gesture action (ST0 → ST6 → ST7 → ST0). The action (B) itself is the same as the first gesture, but if the action (B) is performed immediately after the first gesture and within a certain period of time, it is treated as the second gesture. In other words, the action (B) is a double-tap gesture that combines multiple actions.

[0109] The first gesture and the second gesture are defined to be different from the operator's normal finger movements. This allows the first gesture and the second gesture to be easily detected from any finger movements. In addition, it is preferable that the first gesture is a characteristic finger movement different from the second gesture. This allows the control device 30 to easily distinguish the first gesture from the second gesture. In this embodiment, the first gesture is a large up and down movement of the thumb, and the second gesture is a back and forth and left and right movement of the thumb by the index finger in a plane or a small downward movement based on the same plane.

[0110] Fig. 14 is a diagram showing combinations of peak values ​​of angular velocity of the thumb F1 used in determining each gesture in the operation input system 2 according to this embodiment. A circle ("o") in the diagram indicates a direction in which a peak value of angular velocity equal to or greater than a predetermined threshold value (hereinafter referred to as "peak angular velocity") is detected among the six directions of +X, -X, +Y, -Y, +Z, and -Z. A combination of directions of peak angular velocity marked with a circle indicates a characteristic of each gesture. Note that even when there is no circle, it is assumed that the angular velocity is changing slightly.

[0111] First, a change in the angular velocity of the thumb F1 during the second gesture will be described. During the second gesture of moving the thumb F1 to the right, the angular velocity of the thumb F1 changes significantly in the three directions of +X, +Y, and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the three directions of +X, +Y, and -Z.

[0112] Similarly, during the second gesture of moving the thumb F1 to the left, the angular velocity of the thumb F1 changes significantly in two directions, -Y and +Z. The second gesture of moving the thumb F1 to the left is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, -Y and +Z.

[0113] During the second gesture of moving the thumb F1 forward, the angular velocity of the thumb F1 changes significantly in five directions, +X, +Y, -Y, +Z, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the five directions, +X, +Y, -Y, +Z, and -Z.

[0114] During the second gesture of moving the thumb F1 backward, the angular velocity of the thumb F1 changes significantly in five directions, +X, -X, +Y, -Y, and -Z. The second gesture of moving the thumb F1 backward is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the five directions, +X, -X, +Y, -Y, and -Z.

[0115] During the second gesture of tapping the thumb F1, the angular velocity of the thumb F1 changes significantly in the three directions of +X, -X, and +Y. The second gesture of tapping the thumb F1 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the three directions of +X, -X, and +Y.

[0116] During the second gesture of pushing down the thumb F1, the angular velocity of the thumb F1 changes significantly in two directions, -X and -Z. The second gesture of pushing down the thumb F1 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, -X and -Z.

[0117] Next, a change in the angular velocity of the thumb F1 during the first gesture will be described. As described above, the first gesture is a series of movements of the thumb F1 and index finger F2 from one of states ST0 to ST5 to the initial state ST0 via states ST6 and ST7. For this reason, Fig. 14 separately shows the presence or absence of detection of the peak angular velocity of the thumb F1 during the movement to the released state ST6 and the movement to the initial state ST0.

[0118] During the action of putting the thumb F1 and index finger F2 into the release state ST6, the angular velocity of the thumb F1 changes significantly in the four directions of +X, +Y, +Z, and -Z. Therefore, the action of putting the thumb F1 into the release state ST6 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the four directions of +X, +Y, +Z, and -Z.

[0119] Furthermore, during the movement of the thumb F1 and index finger F2 from the released state ST6 to the initial state ST0, the angular velocity of the thumb F1 changes significantly in two directions, +X and +Y. The movement of the thumb F1 and index finger F2 from the released state ST6 to the initial state ST0 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, +X and +Y.

[0120] 15 is a diagram showing a combination of peak angular velocities of the index finger F2 used in determining each gesture in the operation input system 2 according to this embodiment. The circles ("◯") in the figure are the same as those in FIG.

[0121] First, a change in the angular velocity of the index finger F2 during the second gesture will be described. During the second gesture of moving the thumb F1 to the right, the angular velocity of the index finger F2 changes significantly in two directions, -X and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, -X and -Z.

[0122] During the second gesture of moving the thumb F1 to the left, no peak angular velocity of the index finger F2 is detected in either direction. However, the angular velocity of the index finger F2 changes slightly in either direction. During the second gesture of moving the thumb F1 to the left, no peak angular velocity combination of the index finger F2 is detected, and therefore no feature is used for gesture determination.

[0123] During the second gesture of moving the thumb F1 forward, the angular velocity of the index finger F2 changes significantly in the three directions, -X, -Y, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the three directions, -X, -Y, and -Z.

[0124] During the second gesture of moving the thumb F1 backward, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of moving the thumb F1 backward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, +X and +Y.

[0125] During the second gesture of tapping the thumb F1, no peak angular velocity of the index finger F2 is detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. Since no peak angular velocity combination of the index finger F2 is detected during the second gesture of tapping the thumb F1, there is no feature that can be used for gesture determination.

[0126] During the second gesture of pushing down the thumb F1, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of pushing down the thumb F1 is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, +X and +Y.

[0127] Next, a change in the angular velocity of the index finger F2 during the first gesture will be described. In Fig. 15, similar to Fig. 14, the presence or absence of detection of the peak angular velocity during the action to set the release state ST6 and the action to set the initial state ST0 are separately shown.

[0128] During the action of putting the thumb F1 and index finger F2 into the released state ST6, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the first half of the action of the first gesture of putting the thumb F1 and index finger F2 into the released state ST6, the combination of peak angular velocities of the index finger F2 is not detected, and therefore there is no feature that is used for gesture determination.

[0129] During the movement from the released state ST6 to the initial state ST0, the angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the latter half of the first gesture movement from the released state ST6 to the initial state ST0, the combination of peak angular velocities of the index finger F2 is not detected, so there is no feature used for gesture determination.

[0130] Fig. 16 is a diagram for explaining the relationship between a plurality of gestures detected in the operation input system 2 according to this embodiment. In Fig. 16, eight types of gestures G0 to G6, G8 corresponding to the above-mentioned states ST0 to ST6, ST8 are shown. The direction of the arrow in the figure indicates the direction of transition from one gesture to another. Fig. 17 is a side view for explaining gesture G5. Fig. 18 is a side view for explaining gesture G6.

[0131] Gesture G0 is a first gesture that changes from a state ST6 in which the thumb F1 and index finger F2 are separated to a state ST7 in which the thumb F1 is in contact with the side of the middle phalanx of the index finger F2, and then returns to the initial state ST0, which is maintained for a certain period of time.

[0132] Gesture G1 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is moved leftward along the side of the index finger F2 while maintaining contact between the thumb F1 and index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST1 by gesture G1.

[0133] Gesture G2 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is moved to the right along the side of the index finger F2 while being in contact with the thumb F1 and index finger F2 for a certain period of time. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST2 by gesture G2.

[0134] Gesture G3 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is pushed outward beyond the back of the index finger F2 while the thumb F1 and index finger F2 are in contact with each other. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST3 by gesture G3.

[0135] Gesture G4 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are brought into contact with each other and contracted, and the tip of the thumb F1 is moved to the side of the index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST4 by gesture G4.

[0136] Gesture G5 is a second gesture in which, while the thumb F1 and index finger F2 are in contact with each other when they are in the initial state ST0, the tip of the thumb F1 is pressed down to contact the index finger F2 and middle finger F3 as shown in Fig. 17. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST5 by gesture G5.

[0137] Gesture G6 is a second gesture in which the thumb F1 and index finger F2 are separated from each other from an arbitrary state and maintained in that state for a certain period of time, as shown in Fig. 18. For example, the state of the thumb F1 and index finger F2 is changed from the initial state ST0 or any one of states ST1 to ST5 to the released state ST6 by gesture G6.

[0138] Gesture G8 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are temporarily separated and then the thumb F1 is tapped on the side of the index finger F2 a predetermined number of times (e.g., twice) within a predetermined time limit.

[0139] Gestures G1 to G5 and G8 are second gestures and are associated in advance with six types of commands related to operation input. Gestures G1 to G4 correspond to commands for moving a cursor or the like leftward, rightward, upward, and downward, respectively, on the operation input screen. Gesture G5 corresponds to a command for canceling the operation input performed immediately before on the operation input screen. Gesture G8 corresponds to a command for confirming a selected item or input information on the operation input screen.

[0140] Fig. 19 is a flowchart showing an example of processing executed in the operation input system 2 according to this embodiment. Steps S101 to S108 and S110 to S113 of the processing in Fig. 19 are common to the flowchart in Fig. 10. Therefore, steps different from Fig. 10 will be described in detail below.

[0141] In step S108, the control device 30 determines the relative finger motion of the two fingers based on the combination of the peak values ​​of the angular velocities extracted within the extraction period. The finger motion determination is started when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 is equal to or greater than a predetermined threshold (step S106: YES), thereby reducing the calculation load.

[0142] After step S108, the process proceeds to step S201. In step S201, the control device 30 determines whether or not the finger motion using two fingers corresponds to a first gesture.

[0143] Here, if the control device 30 determines that the finger movement corresponds to the first gesture, i.e., that the finger movement is gesture G0 (step S201: YES), the control device 30 updates the detection flag for the first gesture stored in the memory area to ON (step S202), and the process returns to step S101.

[0144] The detection flag is a flag indicating whether or not the first gesture has been detected. The detection flag may be initialized to OFF when, for example, an operation input command is issued. Note that the method of managing the detection state of the first gesture in the operation input system 1 is not limited to the method of using the detection flag.

[0145] On the other hand, if the control device 30 determines that the finger movement does not correspond to the first gesture (step S201: NO), the process proceeds to step S203.

[0146] In step S203, the control device 30 determines whether the finger motion is the second gesture. If the control device 30 determines that the finger motion is the second gesture (step S203: YES), the process proceeds to step S204.

[0147] On the other hand, if the control device 30 determines that the finger movement is not the second gesture (step S203: NO), the process returns to step S101.

[0148] In step S204, the control device 30 determines whether the detection flag of the first gesture is ON. That is, the control device 30 determines whether the first gesture has been detected before the second gesture. If the control device 30 determines that the detection flag of the first gesture is ON (step S204: YES), the process proceeds to step S205.

[0149] On the other hand, when the control device 30 determines that the detection flag of the first gesture is not ON (step S204: NO), the process returns to step S101. That is, when the control device 30 determines that the second gesture is performed before the first gesture is detected, the control device 30 does not accept the operation input by the second gesture.

[0150] In step S205, the control device 30 outputs a command corresponding to the second gesture. Specifically, when the control device 30 determines that the second gesture has been performed following the first gesture, it accepts the second gesture as an operation input and outputs a command corresponding to the second gesture. Thereafter, the process proceeds to step S110. The processing of steps S110 to S113 is the same as in the case of FIG. 10.

[0151] In the case of FIG. 7 described above, the finger movement from the start time t1 of the extraction period A to the end time t4 of the extraction period B corresponds to the first gesture. Also, the finger movement in the extraction period C following the first gesture corresponds to the second gesture. In the first gesture, a pair of large amplitudes on the plus side and then the minus side occurs for the angular velocity X_1 around the X-axis of the thumb F1, and it is easy to determine the upward finger movement of the thumb F1 in the extraction period A and the downward finger movement of the thumb F1 in the extraction period B. Also, it can be easily determined that the combination of the finger movement in the extraction period A and the finger movement in the extraction period B is the first gesture.

[0152] For example, in FIG. 7, a threshold value THp is set on the plus side and a threshold value THm is set on the minus side for the angular velocity X_1 around the X-axis of the thumb F1. Here, the control device 30 (gesture detection unit 30H) classifies the period in which X_1 ≥ THp as "1", the period in which THm < X_1 < THp as "0", and the period in which X_1 ≤ THm as "-1" for the angular velocity X_1. In this case, the control device 30 can easily detect the angular velocity change pattern of "0" → "1" → "0" → "-1" → "0" in the period from time t1 to t6. Then, the control device 30 performs the detection process of the angular velocity change pattern for the other angular velocities Y_1, Z_1, X_2, Y_2, Z_2 in the same manner as the angular velocity X_1. Thereby, the control device 30 can detect the second gesture performed following the first gesture with a small computational load.

[0153] FIG. 20 is a diagram showing an example of an operation input screen displayed on the display device 306 according to this embodiment. In FIG. 20, a plurality of menus are displayed on the operation input screen, and a state in which a menu M1 is selected from among them is shown. Also, a cross key K for moving the position of the cursor is displayed in the upper right region of the operation input screen. A plurality of indicators are provided around the cross key K. When a command for moving the position of the cursor up, down, left or right is issued by the operator's finger movement, an indicator of the direction corresponding to the command is lit. In FIG. 20, an indicator below the cross key K is lit in response to the issuance of a command for moving the position of the cursor downward.

[0154] Conventional operation input systems based on finger gestures sometimes had difficulty in detecting the boundary between one gesture and the next, which could result in two gestures being detected as one, or the operator's normal finger movements being detected as a gesture, resulting in false detection.

[0155] For example, in a gesture of moving a finger horizontally, if the finger stops almost halfway and then moves further in the same direction, the gesture of horizontal movement may be detected once or may be determined to have been repeated twice in succession. Also, if the operator realizes a mistake midway through the movement and moves in the opposite direction, only one of the gestures may be valid, or both may be valid. In this way, the gesture made by the operator may be detected as a gesture different from the operator's intention.

[0156] Furthermore, conventional operation input systems have the restriction that finger gestures cannot be detected correctly unless the palm or arm is in a specific position (for example, the palm is parallel to the ground). Addressing these issues requires the use of multiple types of sensors and extremely complex calculations.

[0157] In contrast, in the operation input system 2 according to the present embodiment, the operator first performs a first gesture using the thumb F1 and index finger F2, and then performs a second gesture following the first gesture. When the control device 30 detects a combination of the first gesture and the subsequent second gesture, it outputs a command that is associated in advance with the second gesture.

[0158] Therefore, the operation input system 2 according to this embodiment has the advantages of high gesture detection and determination accuracy and low calculation load. Furthermore, since the operation input system 2 is configured to detect the second gesture based on the posture at the time of the first gesture, the operator can input a gesture in any stationary posture, and the degree of freedom during operation is increased.

[0159] [Third embodiment] The operation input system 3 according to the third embodiment will be described below. The following mainly describes the differences from the first and second embodiments, and descriptions of common parts will be omitted or simplified.

[0160] Fig. 21 is a block diagram showing an example of a hardware configuration of devices constituting the operation input system 3 according to this embodiment. As shown in Fig. 21, smart glasses 40 include an MCU 401, a wireless communication device 402, a battery 403, and a display device 404. The operation input system 3 according to this embodiment differs from the first and second embodiments in that it further includes smart glasses 40.

[0161] The smart glasses 40 are a wearable display that can be worn by an operator as glasses. The smart glasses 40 have, for example, an Augmented Reality (AR) function, a camera function, a Bluetooth / Wi-Fi connection function, a microphone function, and the like.

[0162] Fig. 22 is a perspective view showing an example of smart glasses 40 according to this embodiment. Here, the smart glasses 40 are shown to include a spectacle frame 41 and a display device (display unit) 404 that projects a screen onto lenses 42 fitted into the spectacle frame 41. An operation input screen output from the control device 30 as shown in Fig. 20 is projected onto the display device 404.

[0163] The smart glasses 40 according to this embodiment are controlled by the control device 30, for example, in the following manner. In a mode in which only images and videos are displayed, the control device 30 disables operation input by gestures through the OS or application. When the control device 30 switches the execution mode to a mode in which input by an operator is required, the control device 30 enables operation input by gestures through the OS or application.

[0164] This allows the combination of the first gesture and the subsequent second gesture to be detected, making it possible to input operations using gestures. For example, the operator inputs operations using an operation input screen such as that shown in Fig. 20. If the execution mode is then switched to a mode that only displays images and videos, the OS or application will again disable operation inputs using gestures.

[0165] According to the operation input system 3 of this embodiment, an operator wears a ring-shaped device on a thumb F1 and an index finger F2 and performs a predetermined gesture to perform an operation input. An operation input screen is displayed on the smart glasses 40. The operation input screen is updated in real time according to the operation input by the operator's finger gestures.

[0166] For example, when the control device 30 is a smartphone, the operator can perform desired operation input while looking at the screen displayed on the smart glasses 40, not the screen displayed on the smartphone. Therefore, the operator can check the contents of the operation input performed based on the gestures of the first sensor device 10 and the second sensor device 20 through the smart glasses 40, thereby enabling efficient operation input.

[0167] [Modified embodiment] The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0168] For example, the first sensor device 10 and the second sensor device 20 may have an inertial measurement unit as a sensor module, which further includes sensors such as an acceleration sensor and a magnetic sensor in addition to the gyro sensor. Also, the first sensor device 10 and the second sensor device 20 may use other sensors such as an acceleration sensor instead of the gyro sensor as a sensor module.

[0169] Also, instead of the peak value of the angular velocity in the extraction period, an integrated value of the angular velocity in a predetermined period may be used. By identifying the change pattern of the integrated value, it is possible to identify what kind of finger movement has been performed. This also applies to the case where the first sensor device 10 and the second sensor device 20 are configured to detect acceleration instead of angular velocity.

[0170] Moreover, the setting information in the first sensor device 10 and the second sensor device 20 is set according to the finger on which they are to be worn. For this reason, it is preferable that the outer periphery of the base 11 of each of the first sensor device 10 and the second sensor device 20 has characters or figures indicating the finger on which they are to be worn and the direction in which they are to be worn. This allows the operator to wear the first sensor device 10 and the second sensor device 20 in an optimal state on each finger by referring to the characters or figures on the outer periphery of the first sensor device 10 and the second sensor device 20, thereby further improving the accuracy of finger motion detection.

[0171] In the above-described embodiment, the finger motion is determined based on the relationship between the first angular velocity and the second angular velocity detected by the first sensor device 10 and the second sensor device 20 and a predetermined threshold value. However, the method of determining the finger motion is not limited to this. For example, the control device may be configured to determine the finger motion by inputting the detected first signal and second signal to a learning model that has learned finger motions corresponding to the first angular velocity (first signal) and the second angular velocity (second signal). When the learning model is used to determine the finger motion, an effect of further reducing the calculation load is achieved.

[0172] In the above-described embodiment, the description is based on the premise that the posture of the hand is almost maintained during the period from when the operator makes the first gesture to when the operator completes the second gesture. However, if the posture of the operator's hand is significantly different between when the operator makes the first gesture and when the operator makes the second gesture, the accuracy of gesture detection may decrease or the calculation load may increase. Therefore, it is preferable that the control device 30 further includes a configuration for estimating the posture of the hand other than the first finger and the second finger of the operator based on the first angular velocity (first signal) and the second angular velocity (second signal). Specifically, the direction of the palm and the back of the hand is estimated. By accurately estimating the posture of the hand, the accuracy of detection of the first gesture and the second gesture can be improved.

[0173] The control device 30 may determine the second gesture on the condition that the change in the hand posture after the detection of the first gesture is within a predetermined range. This can suppress a decrease in the accuracy of gesture detection. This also has the effect of reducing the calculation load on the control device 30. Furthermore, when the control device 30 determines that the estimated hand posture during the second gesture is significantly different from the hand posture during the first gesture, it is preferable that the control device 30 can provide feedback to that effect to the operator.

[0174] In the above-described embodiment, the first gesture and the second gesture are detected based on the peak value of the angular velocity in the extraction period and a common threshold value. However, the threshold value used for detecting the first gesture may be set to a value different from the threshold value used for detecting the second gesture. For example, the absolute value of the threshold value when detecting the second gesture may be smaller than the absolute value of the threshold value when detecting the first gesture. In this case, the absolute value of the angular velocity of the thumb F1 and the index finger F2 in the second gesture is smaller than the absolute value of the angular velocity of each finger in the first gesture. In addition, since the sensitivity when detecting the second gesture is higher than when detecting the first gesture, the operator can perform the second gesture with a smaller movement than when performing the first gesture after detecting the first gesture. [Explanation of symbols]

[0175] 1, 2, 3... Operation input system 10... First sensor device 20...Second sensor device 30...Control device 30A···Calibration section 30B...Angular velocity calculation section 30C Peak value extraction section 30D Command issuing section 30E···Feedback control section 30F Display control unit 30G···Input section 30H Gesture detector 40. Smart Glasses 101···MCU 102 Sensor module 103 Wireless communication device 104 Feedback Module 105A...Tightening mechanism 105B...Vibration mechanism 105 Battery 201···MCU 202 Sensor module 203 Wireless communication device 204 Feedback Module 205 Battery 301 Processor 302...RAM 303...ROM 304...Storage 305 Communication I / F 306...Display device 307 Input device 401···MCU 402 Wireless communication device 403...Display device 404 Battery

Claims

1. a first sensor device having an annular base body that can be worn on an arbitrary first finger among five fingers of one hand of an operator, and that outputs a first signal in response to a movement of the first finger; a second sensor device having an annular base that can be worn on a second finger different from the first finger among the five fingers and that outputs a second signal corresponding to a movement of the second finger; a control device that determines a relative finger motion of the first finger and the second finger based on the first signal and the second signal; Equipped with Each of the first sensor device and the second sensor device includes a plurality of sensor modules and at least one feedback module that performs a notification operation to the operator, The feedback module is an operation input system having one of a clamping mechanism that clamps the first finger or the second finger and a vibration mechanism that imparts vibration to the first finger or the second finger.

2. the first sensor device includes a first wireless communication device configured to wirelessly transmit the first signal to the control device; the second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication; The operation input system according to claim 1 .

3. Each of the plurality of sensor modules includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor. The operation input system according to claim 1 .

4. The plurality of sensor modules are arranged at substantially equal intervals on the circumference of the base body. The operation input system according to claim 3 .

5. The outer periphery of the base body is provided with letters or figures indicating the finger on which the device is to be worn and the direction in which the device is to be worn. The operation input system according to claim 1 .

6. the control device determines the mounting states of the first sensor device and the second sensor device based on the first signal and the second signal. The operation input system according to claim 1 .

7. the control device performs calibration of the first signal and the second signal at the start-up of the first sensor device and the second sensor device. The operation input system according to claim 1 .

8. The tightening mechanism includes an inflatable or deflated air bag. The operation input system according to claim 1 .

9. The fastening mechanism includes an expandable or contractible dielectric member. The operation input system according to claim 1 .

10. The vibration mechanism has a piezoelectric element that can expand or contract. The operation input system according to claim 1 .

11. The vibration mechanism has an electromagnetic drive unit capable of vibrating. The operation input system according to claim 1 .

12. The operation input system according to claim 1 , wherein each of the first sensor device and the second sensor device includes a plurality of feedback modules corresponding to the plurality of sensor modules, respectively.

13. a first sensor device that is attached to an arbitrary first finger among five fingers of one hand of an operator and detects a first signal corresponding to a movement of the first finger; a second sensor device that is attached to a second finger, different from the first finger, among the five fingers and detects a second signal corresponding to a movement of the second finger; A control device that controls each of the first sensor device and the second sensor device; Equipped with Each of the first sensor device and the second sensor device includes a plurality of sensor modules and at least one feedback module that performs a notification operation to the operator, the feedback module has one of a clamping mechanism that clamps the first finger or the second finger and a vibration mechanism that imparts vibration to the first finger or the second finger; The control device includes: Detecting a relative finger motion of the first finger and the second finger based on the first signal and the second signal; on a condition that the finger motion is determined to be a predetermined first gesture, determining whether or not the finger motion performed following the first gesture is a predetermined second gesture; When the finger motion is determined to be the second gesture, a command corresponding to the second gesture is output. Operation input system.

14. the control device starts determining the finger motion when the first signal or the second signal exceeds a threshold. The operation input system according to claim 13.

15. The threshold value for determining the first gesture is greater than the threshold value for determining the second gesture. The operation input system according to claim 14.

16. the control device estimates a posture of a hand of the operator other than the first finger and the second finger based on the first signal and the second signal. The operation input system according to claim 13.

17. the control device determines the second gesture on the condition that a change in the posture of the hand after the detection of the first gesture is within a predetermined range. The operation input system according to claim 16.

18. the control device determines the finger motion by inputting the detected first signal and the second signal into a learning model that has learned the finger motion corresponding to the first signal and the second signal. The operation input system according to claim 13.

19. The first finger is a thumb and the second finger is an index finger, The first gesture is an action of moving the thumb and the index finger apart and then touching the thumb to a side surface of the index finger. The operation input system according to claim 13.

20. The second gesture is an action of moving the thumb leftward, rightward, forward, and backward along the side surface of the index finger while the thumb and the index finger are in contact with each other. The operation input system according to claim 19.

21. The second gesture is an action of touching the thumb and the index finger, pressing down the thumb, and touching the thumb to the index finger and the middle finger. The operation input system according to claim 19.

22. the second gesture is an action of moving the thumb and the index finger apart from each other; The operation input system according to claim 19.

23. The second gesture is an action of touching the thumb to the index finger after consecutively shaking the thumb up and down a predetermined number of times. The operation input system according to claim 19.

24. the control device outputs the command to move a position of a pointer or a cursor displayed on an operation input screen to any one of a leftward direction, a rightward direction, an upward direction, and a downward direction in response to the second gesture. The operation input system according to claim 19.

25. the control device outputs, in response to the second gesture, the command for canceling an operation input performed immediately before on an operation input screen. The operation input system according to claim 19.

26. the control device outputs, in response to the second gesture, the command to reset a flag indicating a detection state of the first gesture. The operation input system according to claim 19.

27. the control device outputs, in response to the second gesture, the command for determining a previously performed operation input on an operation input screen. The operation input system according to claim 19.

28. the first sensor device includes a first wireless communication device configured to wirelessly transmit the first signal to the control device; the second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication; The operation input system according to claim 19.

29. Each of the plurality of sensor modules includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor. The operation input system according to claim 19.

30. a wearable display including a spectacle frame and a display unit that projects an operation input screen output from the control device onto lenses fitted in the spectacle frame; The operation input system according to claim 19, further comprising:

Images