Operation input system

Abstract

To provide an operation input system allowing an operator to perform input operation accurately by a finger.SOLUTION: An operation input system according to the present invention includes: a first sensor device which has an annular base body being wearable to a first finger of an operator, and outputs a first signal according to a motion of the first finger; a second sensor device which has an annular base body being wearable to a second finger different from the first finger and outputs a second signal according to a motion of the second finger; and a control device which determines a relative finger motion of the first finger and the second finger on the basis of the first signal and the second signal.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an operation input system.

Background Art

[0002] Patent Document 1 discloses a smart interface ring having a pressure-sensitive touch pointer and a three-axis gyro sensor, which is worn on one finger of an operator. This smart interface ring captures the movement of the fingertip based on the angular velocity of the finger detected by the three-axis gyro sensor, enabling operation input to an application.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the ring described in Patent Document 1, since it is configured to detect only the angular velocity of one finger of a hand in a predetermined posture state and determine the movement, the finger movement of the operator cannot be accurately determined, and there is a problem that the accuracy of the input operation is low.

[0005] Therefore, in view of the problems in the above-described prior art, an object of the present invention is to provide an operation input system that enables an operator to perform an input operation by a finger movement with high accuracy.

Means for Solving the Problems

[0006] According to one aspect of the present invention, there is provided an operation input system including: a first sensor device having an annular base body attachable to a first finger of an operator and outputting a first signal corresponding to the movement of the first finger; a second sensor device having an annular base body attachable to a second finger different from the first finger and outputting a second signal corresponding to the movement of the second finger; and a control device configured to determine a relative finger movement between the first finger and the second finger based on the first signal and the second signal.

[0007] According to another aspect of the present invention, there is provided an operation input system including: a first sensor device attached to a first finger of an operator and configured to detect a first signal corresponding to the movement of the first finger; a second sensor device attached to a second finger different from the first finger and configured to detect a second signal corresponding to the movement of the second finger; and a control device configured to control the first sensor device and the second sensor device, wherein the control device is configured to detect a relative finger movement between the first finger and the second finger based on the first signal and the second signal, and to determine whether a finger movement following the first gesture is a predetermined second gesture on the condition that the finger movement is determined to be a predetermined first gesture, and to output a command corresponding to the second gesture when the finger movement is determined to be the second gesture.

Advantages of the Invention

[0008] According to the present invention, there is provided an operation input system that enables an operator to perform an input operation by finger movement with high accuracy.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. Elements having common functions throughout the drawings are denoted by the same reference numerals, and duplicate explanations may be omitted or simplified.

[0011] [First Embodiment] FIG. 1 is a block diagram showing an example of the hardware configuration of the device constituting the operation input system 1 according to the present embodiment.

[0012] As shown in FIG. 1, the operation input system 1 includes a first sensor device 10, a second sensor device 20, and a control device 30. In the present embodiment, the first sensor device 10 and the second sensor device 20 are connected to the control device 30 via a network.

[0013] The first sensor device 10 has an annular base that can be worn on the first finger of the operator, and outputs a first signal corresponding to the movement of the first finger. The first sensor device 10 of the present embodiment outputs detection data of the angular velocity of the first finger (hereinafter referred to as "first angular velocity") as a first signal corresponding to the movement of the first finger. Specifically, the first sensor device 10 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

[0014] As shown in FIG. 1, the first sensor device 10 includes an MCU (Micro Controller Unit) 101, a sensor module 102, a feedback module 103, a wireless communication device 104, and a battery 105. Each part of the first sensor device 10 is connected by wiring on a circuit board (not shown).

[0015] The MCU 101 is an embedded microprocessor that integrates a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an I / O device, etc. into one integrated circuit. The MCU 101 is mainly used for controlling electronic devices.

[0016] The sensor module 102 has a gyro sensor that detects angular velocity. The gyro sensor of this embodiment is a three-axis gyro sensor that detects the angular velocity around the X-axis, Y-axis, and Z-axis that are orthogonal to each other. Note that the sensor module 102 may further include sensors such as an acceleration sensor and a magnetic sensor in addition to the gyro sensor. When an acceleration sensor is used, the actual acceleration is calculated by removing the component of gravity from the detected acceleration value.

[0017] The sensor module 102 applies a voltage proportional to the detected angular velocity to the MCU 101. The MCU 101 performs A / D conversion of the voltage applied from the sensor module 102 into digital data. Then, the MCU 101 transmits the detection data of the angular velocity to the control device 30 via the wireless communication device 104 described later.

[0018] The feedback module 103 gives a notification to the operator wearing the first sensor device 10 on the thumb based on the control information received from the control device 30. Details of the feedback module 103 will be described later.

[0019] The wireless communication device 104 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for performing wireless communication with other devices. The wireless communication device 104 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

[0020] The battery 105 is a device that supplies driving power to each of the MCU 101, the sensor module 102, the feedback module 103, and the wireless communication device 104.

[0021] The second sensor device 20 has an annular substrate that can be worn on a second finger different from the first finger of the operator, and outputs a second signal corresponding to the movement of the second finger. The second sensor device 20 of the present embodiment outputs the detection data of the angular velocity of the second finger (hereinafter referred to as "second angular velocity") as a second signal corresponding to the movement of the second finger. Specifically, the second sensor device 20 transmits the detection data of the second angular velocity to the control device 30 by wireless communication.

[0022] In addition, in the present embodiment, the first finger is the thumb. Since it is preferable that the second finger is a finger adjacent to the first finger, the second finger is the index finger. In addition, it is assumed that the setting information in the first sensor device 10 and the second sensor device 20 is set according to the thumb and index finger of the wearing target.

[0023] As shown in FIG. 1, the second sensor device 20 includes an MCU 201, a sensor module 202, a feedback module 203, a wireless communication device 204, and a battery 205. Since the hardware configuration of the second sensor device 20 is the same as the hardware configuration of the first sensor device 10, the description of each device is omitted.

[0024] The control device 30 is a computer device that performs calculation, control, and storage. Examples of the control device 30 include a personal computer, a laptop computer, a tablet terminal, and a smartphone. The control device 30 determines the relative finger movements of the first finger and the second finger based on the first angular velocity and the second angular velocity.

[0025] As shown in FIG. 1, the control device 30 includes a processor 301, a RAM 302, a ROM 303, a storage 304, a communication I / F (Interface) 305, a display device 306, and an input device 307. Each device is interconnected via a bus, wiring, a driving device, or the like.

[0026] The processor 301 has a function of performing a predetermined calculation according to programs stored in the ROM 303, the storage 304, etc., and controlling each part of the control device 30, the first sensor device 10, and the second sensor device 20. As the processor 301, a CPU, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like is used.

[0027] <000## **Answer**:

[0025] As shown in FIG. 1, the control device 30 includes a processor 301, a RAM 302, a ROM 303, a storage 304, a communication I / F (Interface) 305, a display device 306, and an input device 307. Each device is interconnected via a bus, wiring, a driving device, or the like.

[0026] The processor 301 has a function of performing a predetermined calculation according to programs stored in the ROM 303, the storage 304, etc., and controlling each part of the control device 30, the first sensor device 10, and the second sensor device 20. As the processor 301, a CPU, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like is used.

[0027] The RAM 302 is a storage device composed of a volatile storage medium and provides a temporary memory area necessary for the operation of the processor 301. The RAM 302 may be, for example, a D-RAM (Dynamic RAM).

[0028] The ROM 303 is a storage device composed of a non-volatile storage medium and stores necessary information such as programs used for the operation of the control device 30. The ROM 303 may be, for example, a P-ROM (Programmable ROM).

[0029] The storage 304 is a storage device composed of a non-volatile storage medium and stores processing data in the control device 30, operation programs of the control device 30, and the like. The storage 304 is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0030] The processor 301 loads and executes programs stored in the ROM 303, the storage 304, etc. into the RAM 302.

[0031] The communication I / F 305 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for performing wireless communication with the first sensor device 10 and the second sensor device 20.

[0032] The display device 306 displays videos, still images, characters, etc. As the display device 306, a liquid crystal display, an OLED (Organic Light Emitting Diode) display, etc. are used. The operation input screen displayed on the display device 306 includes graphics such as windows, icons, buttons, etc. The operator selects the graphics representing the target operation in the operation input screen by means of operation input means using a pointing device, or operation input means using the first sensor device 10 and the second sensor device 20.

[0033] The input device 307 is a keyboard, a pointing device, etc., and is used for the operator to operate the control device 30. Examples of the pointing device include a mouse, a trackball, a touch panel, a tablet, etc. The display device 306 and the input device 307 may be integrally formed as a touch panel.

[0034] Note that the hardware configuration shown in FIG. 1 is an example, and devices other than these may be added, or some devices may not be provided. Also, some devices may be replaced with other devices having similar functions. Further, some functions of the first embodiment may be provided by other devices via a network, or the functions of the first embodiment may be realized by being distributed among a plurality of devices. The illustrated hardware configuration can be changed as appropriate.

[0035] FIG. 2 is a diagram showing an example of attachment of the first sensor device 10 and the second sensor device 20 according to the present embodiment to an operator's finger. In FIG. 2, the first sensor device 10 is attached to the proximal end side of the proximal phalanx of the operator's thumb F1. The second sensor device 20 is attached to the proximal end side of the proximal phalanx of the operator's index finger F2.

[0036] Also, in the XYZ coordinate system of the second sensor device 20, the X-axis is set in the width direction of the operator's index finger F2, the Y-axis is set in the direction in which the proximal phalanx of the index finger F2 extends, and the Z-axis is set in the thickness direction of the index finger F2. The X-axis, Y-axis, and Z-axis are orthogonal to each other. Note that, similarly to the second sensor device 20, in the XYZ coordinate system of the first sensor device 10, the X-axis is set in the width direction of the operator's thumb F1, the Y-axis is set in the direction in which the proximal phalanx of the thumb F1 extends, and the Z-axis is set in the thickness direction of the thumb F1.

[0037] FIGS. 3 and 4 are cross-sectional views showing an example of the internal structure of the first sensor device 10 according to the present embodiment. Note that since the internal structure of the second sensor device 20 is the same as that of the first sensor device 10, the first sensor device 10 will be described as a representative example below.

[0038] In FIG. 3, an example is shown in which a tightening mechanism 103A is used as the feedback module 103 in the first sensor device 10. The tightening mechanism 103A expands or contracts based on control information from the control device 30. Examples of the tightening mechanism 103A include an airbag or a dielectric member (e.g., next-generation rubber material e-Rubber manufactured by Toyoda Gosei Co., Ltd.) that can expand or contract according to the applied voltage. When the tightening mechanism 103A expands, it deforms into a protruding shape toward the operator's finger, and a pressing force is applied to the operator's finger. Thereby, the control device 30 causes the feedback module 103 to execute feedback on the operation input from the operator. The feedback is performed to notify the operator that the input operation of the operator has been received and a predetermined command has been executed.

[0039] In FIG. 3, the tightening mechanism 103A is formed in a ring shape, similar to the base body 11, but the shape of the tightening mechanism 103A is not limited to this. The tightening mechanism 103A may be formed, for example, in a sheet shape. Also, the number of the tightening mechanisms 103A is one, but it is not limited to this.

[0040] When a plurality of tightening mechanisms 103A are provided, there will be a plurality of locations in the apparatus that can expand or contract. For this reason, by switching the tightening mechanism 103A to be the object of expansion or contraction in the first sensor device 10 according to time, feedback in the clockwise direction, counterclockwise direction, etc. becomes possible. That is, it becomes possible to further increase the variations in the feedback method.

[0041] Also, the first sensor device 10 includes three sensor modules 102. The three sensor modules 102 are arranged at substantially equal intervals on the circumference of the base body 11 in a cross-sectional view. Sensors such as gyro sensors have different changes in the detected angular velocity values depending on the mounting position. For this reason, by providing the three sensor modules 102 apart from each other, the angular velocity can be detected at different positions. As a result, the control device 30 can accurately determine the mounting state of the first sensor device 10 on the thumb F1 based on a plurality of angular velocities. This is the same in the case of the second sensor device 20.

[0042] On the other hand, FIG. 4 shows an example in which three vibration mechanisms 103B are used as the feedback module 103 in the first sensor device 10. The number of the sensor modules 102 is three, the same as in the case of FIG. 3.

[0043] The vibration mechanism 103B vibrates based on control information from the control device 30. The vibration mechanism 103B is composed of, for example, a piezoelectric element (piezo element) that can expand or contract according to the applied voltage, or an electromagnetic drive unit such as a vibratable vibration motor. The piezo element is a passive element that performs micro motion control and detection by utilizing the piezoelectric effect and inverse piezoelectric effect that occur in dielectrics such as quartz and quartz. Since it has a simple structure that does not require gears or motors for operation, it is a small element compared to other micro motion mechanism elements.

[0044] Also, the vibration mechanism 103B does not have to be provided adjacent to the sensor module 102, and may be provided between a plurality of sensor modules 102. When the vibration mechanism 103B vibrates, the vibration is transmitted to the operator's finger. Thereby, the control device 30 causes the feedback module 103 to execute feedback on the operation input from the operator.

[0045] In FIG. 4, the number of vibration mechanisms 103B is the same as the number of sensor modules 102, but the number of vibration mechanisms 103B is not limited to this. The magnitude of vibration, the period of vibration, etc. in each module may be changed.

[0046] When a plurality of vibration mechanisms 103B are provided, there will be a plurality of locations where vibration is possible within the device. By switching the vibration mechanism 103B that vibrates within the first sensor device 10 according to time, feedback in the clockwise, counterclockwise, etc. directions becomes possible. That is, it becomes possible to further increase the variations in the feedback method.

[0047] FIG. 5 is a functional block diagram showing an example of the functions of the control device 30 according to the present embodiment. The control device 30 includes a calibration unit 30A, an angular velocity calculation unit 30B, a peak value extraction unit 30C, a command issuance unit 30D, a feedback control unit 30E, a display control unit 30F, and an input unit 30G.

[0048] When the calibration unit 30A is first activated with the first sensor device 10 and the second sensor device 20 attached to the operator's finger, calibration is performed on the detection data of the angular velocity detected by the sensor modules 102 and 202. The calibration unit 30A outputs the correction value (offset value) of the angular velocity calculated by calibration to the angular velocity calculation unit 30B.

[0049] Based on the detection data of the angular velocity acquired wirelessly from the first sensor device 10 and the second sensor device 20 and the correction value calculated by the calibration unit 30A, the angular velocity calculation unit 30B calculates the angular velocity of the operator's thumb F1 and index finger F2. The angular velocity calculation unit 30B outputs the calculated angular velocity to the peak value extraction unit 30C. The angular velocity calculation process in the angular velocity calculation unit 30B is repeatedly executed each time detection data of the angular velocity is received from the first sensor device 10 and the second sensor device 20.

[0050] The peak value extraction unit 30C extracts the peak value from the angular velocities within a predetermined extraction period. The peak value extraction unit 30C outputs the extracted peak value to the command issuing unit 30D.

[0051] FIG. 6 is a diagram showing an example of the time change of the angular velocity detected by the first sensor device 10 according to the present embodiment and the extraction period of the peak value. In FIG. 6, as shown by the broken line, the positive side threshold THp of the angular velocity is set to +20 rad / sec, and the negative side threshold THm is set to -20 rad / sec. The absolute values of the thresholds THp and THm are equal. Also, the extraction period T is from the time t1 when at least one of the angular velocities around the X-axis, Y-axis, and Z-axis of the two fingers becomes equal to or greater than a predetermined threshold to the time t2 when a certain time has elapsed. However, the values of the thresholds THp and THm and the setting method of the extraction period T are not limited to this.

[0052] The command issuing unit 30D determines the relative finger movements of the two fingers of the operator based on the peak values extracted within the extraction period, and outputs the commands pre-associated with the finger movements. The command issuing unit 30D is assumed to detect at least two types of finger movements. When the command issuing unit 30D outputs a command, a process corresponding to the command is executed in any of the devices constituting the operation input system 1.

[0053] The feedback control unit 30E transmits control information to the MCU 101 of the first sensor device 10 and the MCU 201 of the second sensor device 20, and causes the feedback modules 103 and 203 to execute predetermined feedback processing.

[0054] The display control unit 30F performs display control of the operation input screen displayed on the display device 306 based on the input operation by the input device 307 received by the input unit 30G and the input operations by the first sensor device 10 and the second sensor device 20.

[0055] The input unit 30G receives the information input by the user such as the operator or the administrator using the input device 307 to operate the control device 30.

[0056] FIG. 7 is a diagram showing the time change of the angular velocity of the thumb F1 detected in the first sensor device 10 according to the present embodiment. On the other hand, FIG. 8 is a diagram showing the time change of the angular velocity of the index finger F2 detected in the second sensor device 20 according to the present embodiment. In FIGS. 7 and 8, the extraction period A from time t1 to t2, the extraction period B from time t3 to t4, and the extraction period C from time t5 to t6 are shown.

[0057] Also, the threshold values THp and THm for the angular velocity of the thumb F1 in FIG. 7 are different from the threshold values THp and THm for the angular velocity of the index finger F2 in FIG. 8. Thus, it is preferable that the threshold values THp and THm are set to optimal values for each finger. The control device 30 determines the relative finger movements by the thumb F1 and the index finger F2 in each of the extraction periods A, B, and C based on the peak values of the angular velocities of the two fingers around the X, Y, and Z axes and the threshold values THp and THm in the extraction periods A, B, and C.

[0058] As shown in FIG. 7, in the extraction period A, the angular velocity X_1 of the thumb F1 around the X axis is displaced on the positive side, and the peak value on the positive side is larger than the positive-side threshold value THp. Also, the angular velocity Y_1 of the thumb F1 around the Y axis is displaced on the negative side, and the peak value on the negative side is larger than the negative-side threshold value THm. The angular velocity Z_1 of the thumb F1 around the Z axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp.

[0059] Also, as shown in FIG. 8, in the extraction period A, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THp. Also, the angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on both the positive and negative sides, the peak value on the positive side is smaller than the threshold value THp, and the peak value on the negative side is larger than the threshold value THm.

[0060] Analyzing the changes in the angular velocity in FIGS. 7 and 8 for the extraction period A, it can be seen that the thumb F1 moved significantly in the +X direction around the X axis. Also, in the extraction period A, it can be seen that the index finger F2 moved significantly in the +Y direction around the Y axis. As a result, in the extraction period A, a finger movement that swings the thumb F1 upward from the index finger F2 is detected.

[0061] As shown in FIG. 7, during the extraction period B, the angular velocity X_1 of the thumb F1 around the X-axis is displaced to the negative side, and the peak value on the negative side is smaller than the threshold value THm. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced to the positive side, and the peak value on the positive side is smaller than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced to the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0062] Also, as shown in FIG. 8, during the extraction period B, the angular velocity X_2 of the index finger F2 around the X-axis is displaced to the negative side, and the peak value on the negative side is larger than the threshold value THm. Also, the angular velocity Y_2 of the index finger F2 around the Y-axis is displaced to the negative side, and the peak value on the negative side is smaller than the threshold value THm. The angular velocity Z_2 of the index finger F2 around the Z-axis is displaced to the positive side, and the peak value on the positive side is smaller than the threshold value THp.

[0063] Analyzing the changes in the angular velocity in FIGS. 7 and 8 for the extraction period B, it can be seen that the thumb F1 moved significantly in the -X direction around the X-axis. Also, during the extraction period B, it can be seen that the index finger F2 moved significantly in the -Y direction around the Y-axis. As a result, during the extraction period B, a finger movement that brings the thumb F1 into contact with the index finger F2 is detected.

[0064] As shown in FIG. 7, during the extraction period C, the angular velocity X_1 of the thumb F1 around the X-axis is displaced to the positive side, and the peak value on the positive side is smaller than the threshold value THp. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced to the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced to the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0065] Also, as shown in FIG. 8, during the extraction period C, the angular velocity X_2 of the index finger F2 around the X-axis is displaced to the negative side, and the peak value on the negative side is smaller than the threshold value THm. Also, the angular velocity Y_2 of the index finger F2 around the Y-axis is displaced to the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z-axis is displaced to the negative side, and the peak value on the negative side is larger than the threshold value THm.

[0066] Analyzing the changes in the angular velocity in FIGS. 7 and 8 for the extraction period C, it can be seen that the thumb F1 moved significantly in the +Y direction around the Y-axis. Also, in the extraction period C, it can be seen that the index finger F2 moved significantly in the -X direction around the X-axis and in the +Y direction around the Y-axis, respectively. As a result, in the extraction period C, while the thumb F1 maintains contact with the index finger F2, a finger movement is detected in which the thumb F1 slides to the right along the side surface of the index finger F2.

[0067] FIG. 9 is a timing chart showing the processes executed in the control device 30 according to the present embodiment and the execution times. Here, the correspondence between the extraction periods A, B, and C shown in FIGS. 7 and 8 and the command issuance process, the feedback destination determination process, the drive control process for the feedback module, and the display control process for the operation input screen is shown.

[0068] The first command issuance process in the command issuance unit 30D is executed during the extraction period A from time t1 to t2. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t2 to t21. Then, the drive control process for the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t21 to t22.

[0069] Further, the second command issuing process in the command issuing unit 30D is executed during the extraction period B from time t3 to t4. Next, the process of determining the feedback destination in the feedback control unit 30E is executed during the period from time t4 to t41. Then, the drive control process of the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t41 to t42.

[0070] Then, the third command issuing process in the command issuing unit 30D is executed during the extraction period C from time t5 to t6. Next, the process of determining the feedback destination in the feedback control unit 30E is executed during the period from time t6 to t61. Then, the drive control process of the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t61 to t62.

[0071] FIG. 10 is a flowchart showing an example of the process executed in the control device 30 according to the present embodiment. This process is repeatedly executed, for example, while the operation input screen is being displayed.

[0072] First, the control device 30 acquires angular velocity detection data from the first sensor device 10 and the second sensor device 20 by wireless communication (step S101).

[0073] Next, the control device 30 determines whether it is the first startup of the first sensor device 10 and the second sensor device 20 (step S102). Here, when the control device 30 determines that it is the first startup (step S102: YES), the process proceeds to step S103. On the other hand, when the control device 30 determines that it is not the first startup (step S102: NO), the process proceeds to step S105.

[0074] In step S103, the control device 30 performs calibration based on the detected data of the angular velocity acquired from the first sensor device 10 and the second sensor device 20, and calculates a correction value of the angular velocity. Thereafter, the process proceeds to step S104.

[0075] In step S104, the control device 30 determines the mounting state of the sensor based on the detected data of the angular velocity. Thereafter, the process proceeds to step S105.

[0076] As shown in FIG. 3, the first sensor device 10 includes a plurality of sensor modules 102. Therefore, the control device 30 can determine the mounting state of the first sensor device 10 on the thumb F1 based on the plurality of detected data detected in the plurality of sensor modules 102.

[0077] Similarly, the second sensor device 20 includes a plurality of sensor modules 202. The control device 30 can determine the mounting state of the second sensor device 20 on the index finger F2 based on the plurality of detected data detected in the plurality of sensor modules 202.

[0078] In step S105, the control device 30 calculates the angular velocities of the thumb F1 and the index finger F2 based on the detected data of the angular velocity and the correction value of the angular velocity. The angular velocities of the thumb F1 and the index finger F2 are calculated for the three axes of the X-axis, Y-axis, and Z-axis.

[0079] Next, the control device 30 determines whether any of the absolute values of the calculated plurality of angular velocities is equal to or greater than a predetermined threshold (step S106).

[0080] Here, when the control device 30 determines that the absolute value of any of the angular velocities is equal to or greater than the predetermined threshold (step S106: YES), the process proceeds to step S107. On the other hand, when the control device 30 determines that the absolute values of all the angular velocities are less than the predetermined threshold (step S107: NO), the process returns to step S101.

[0081] In step S107, the control device 30 determines the extraction period of the angular velocity and extracts the peak value from the angular velocities within the extraction period. The peak values are extracted for each of the three axes of the X-axis, Y-axis, and Z-axis. Note that if the angular velocity fluctuates within the extraction period, multiple peak values may be extracted in the same direction. In such a case, the control device 30 sequentially updates, for example, the peak values held in the storage area.

[0082] Next, the control device 30 determines the relative finger movement by the two fingers based on the combination of the peak values of the angular velocities of the two fingers extracted within the extraction period (step S108). Since the determination of the finger movement starts when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 exceeds a predetermined threshold value, the calculation load is suppressed.

[0083] In step S108, for example, finger movements such as the fingers of the thumb F1 and the index finger F2 moving apart, the fingers of the thumb F1 and the index finger F2 coming into contact, and the thumb F1 moving up and down, left and right, or forward and backward along the index finger F2 while the thumb F1 and the index finger F2 are in contact are determined.

[0084] Next, when the control device 30 determines the command corresponding to the relative movement by the two fingers, it outputs the command (step S109). Thereby, the control device 30 executes the process specified by the command. Examples of the command include a command to move the position of the cursor or pointer on the operation input screen up, down, left, or right, a command to cancel the previous operation input, and a command to determine the operation input.

[0085] Next, the control device 30 determines the feedback destination in accordance with the execution of the issued command (step S110). In the present embodiment, the first sensor device 10, the second sensor device 20, and the display device 106 can be the feedback destinations.

[0086] Next, the control device 30 transmits control information to the feedback modules 103 and 203, which are the determined feedback destinations, and drives the feedback modules 103 and 203 (step S111).

[0087] Next, the control device 30 updates the operation input screen displayed on the display device 106 (step S112). For example, the control device 30 changes the coordinates of the pointer on the operation input screen or changes the display state of the selected menu. In this way, the control device 30 notifies the GUI (display device) in synchronization with the notification to the feedback module. Then, the process ends.

[0088] Then, the control device 30 determines whether or not an end of operation input has been instructed by the operator on the operation input screen (step S113). Here, when the control device 30 determines that the end of operation input has been instructed (step S113: YES), the process ends. On the other hand, when the control device 30 determines that the end of operation input has not been instructed (step S113: NO), the process returns to step S101.

[0089] Conventionally, as input means to replace mouse operations, input means using gestures or controllers, input means using eye gaze, input means using voice, etc. are known. However, in the input means using gestures or controllers, the operation input motion becomes large and the operation state is known to others. Also, in the input means using eye gaze, since the eye gaze position fluctuates, stable operation is difficult. And in the input means using voice, the voice instructing the operation is heard by others. In view of these problems, there has been a demand for a pointing device that is difficult to be recognized by others and has excellent operability.

[0090] On the other hand, according to the operation input system 1 according to the present embodiment, a first sensor device 10 and a second sensor device 20, which are ring-shaped devices equipped with gyro sensors, are attached to the thumb (first finger) F1 and the index finger (second finger) F2, respectively. Then, the first sensor device 10 and the second sensor device 20 transmit the angular velocities of the two fingers to the control device 30 by wireless communication. The control device 30 detects the relative finger movements of the two fingers using the angular velocities of the thumb F1 and the index finger F2.

[0091] Thereby, a command corresponding to the finger movement can be determined and output, and operation input can be performed freely and with high precision. In addition, since the configuration allows operation input by simply moving the thumb F1 and the index finger F2, it is difficult for others to know that the operator is performing operation input, which is beneficial from the viewpoints of improving information security and protecting privacy.

[0092] Also, in a conventional system in which a ring-shaped device is attached to only one finger, it is necessary to change the direction of the arm to indicate a specific position with the finger. On the other hand, according to the operation input system 1 according to the present embodiment, by attaching ring-shaped devices equipped with gyro sensors to the thumb F1 and the index finger F2 respectively, it is configured to determine the relative movement of the two fingers. Therefore, pointing is possible in any orientation of the arm.

[0093] Also, in a conventional ring-shaped device, there is one known to be equipped with a gyro sensor for motion detection. However, since the information that can be detected changes depending on the mounting position of the gyro sensor, the ring-shaped device always needs to be mounted in a certain direction. On the other hand, according to the operation input system 1 of the present embodiment, the ring-shaped devices (first sensor device 10 and second sensor device 20) are equipped with a plurality of sensor modules at different positions within the ring. Since the mounting state of the ring-shaped device can be determined based on a plurality of detection data from the plurality of sensor modules, the ring-shaped device can be mounted at an arbitrary position on each finger.

[0094] [Second Embodiment] Hereinafter, the operation input system 2 according to the second embodiment will be described. Hereinafter, mainly the differences from the first embodiment will be described, and the common parts will be omitted or simplified in the description.

[0095] FIG. 11 is a block diagram showing an example of the hardware configuration of the operation input system 2 according to the present embodiment. The hardware configuration of the operation input system 2 is common to the hardware configuration of the operation input system 1 according to the first embodiment.

[0096] FIG. 12 is a functional block diagram showing an example of the functions of the control device 30 according to the present embodiment. Different from the case of the first embodiment, the control device 30 of the present embodiment further includes a gesture detection unit 30H.

[0097] The gesture detection unit 30H detects a specific finger movement from among arbitrary finger movements by the thumb F1 and the index finger F2 based on the angular velocities of the thumb F1 and the index finger F2. In the present embodiment, the specific finger movement that the gesture detection unit 30H detects as a detection target from among arbitrary finger movements is called a "gesture". Note that the gesture is not limited to only a single finger movement. There may be a case where a plurality of finger movements continuously performed within a certain time are defined as one gesture. And it is assumed that the correspondence between a plurality of gestures and the commands issued for each gesture is defined in advance. The command issuing unit 30D of the present embodiment outputs a command corresponding to the specific gesture detected by the gesture detection unit 30H.

[0098] FIG. 13 is a diagram showing the relationship between finger movements and finger state transitions in the operation input system 2 according to the present embodiment. Here, as finger movements, "raising the thumb", "lowering the thumb", "pressing down the thumb", "stationary", and "moving" are shown. Also, as finger states, "initial (ST0)", "left (ST1)", "right (ST2)", "front (ST3)", "rear (ST4)", "down (ST5)", "release (ST6)", and "contact (ST7)" are shown.

[0099] "Initial (ST0)" is a state in which the operator lightly holds the hand and places the thumb F1 at the intermediate position between the first joint and the second joint of the index finger F2. Hereinafter, "Initial (ST0)" will be referred to as the initial state ST0.

[0100] "Left (ST1)" is a state in which, while keeping the thumb F1 and the index finger F2 in contact in the initial state ST0, the tip of the thumb F1 is moved leftward along the side surface of the index finger F2. Hereinafter, "Left (ST1)" will be referred to as the state ST1.

[0101] "Right (ST2)" is a state in which, while keeping the thumb F1 and the index finger F2 in contact in the initial state ST0, the tip of the thumb F1 is moved rightward along the side surface of the index finger F2. Hereinafter, "Right (ST2)" will be referred to as the state ST2.

[0102] "Forward (ST3)" is a state in which, while keeping the thumb F1 and the index finger F2 in contact in the initial state ST0, the tip of the thumb F1 protrudes outward from the index finger F2 toward the base joint. Hereinafter, "Forward (ST3)" will be referred to as the state ST3.

[0103] "Backward (ST4)" is a state in which, while keeping the thumb F1 and the index finger F2 in contact in the initial state ST0, the tip of the thumb F1 is moved to the side surface of the base joint of the index finger F2. Hereinafter, "Backward (ST4)" will be referred to as the state ST4.

[0104] "Down (ST5)" is a state in which, while keeping the thumb F1 and the index finger F2 in contact in the initial state ST0, the tip of the thumb F1 is pushed downward to the lower position of the index finger F2. Hereinafter, "Down (ST5)" will be referred to as the state ST5. In the state ST5, the thumb F1 is positioned between the index finger F2 and the middle finger F3.

[0105] "Release (ST6)" is a state when the in-contact thumb F1 and index finger F2 are separated and maintained for a certain period of time. Hereinafter, "Release (ST6)" will be referred to as the release state ST6.

[0106] "Contact (ST7)" is a state in which the thumb F1 and the index finger F2, which were in the release state ST6, are in contact. Hereinafter, "Contact (ST7)" will be referred to as state ST7. State ST7 is an intermediate state when changing from the release state ST6 to the initial state ST0.

[0107] The control device 30 (gesture detection unit 30H) determines the states of the thumb F1 and the index finger F2 based on the change in angular velocity. The control device 30 determines that the current state is the initial state ST0 when the state transitions from the release state ST6 to the contact state ST7 and the contact state ST7 continues for a certain period of time.

[0108] Also, in the present embodiment, the first gesture and the second gesture are defined as follows. (1) First gesture The first gesture is an operation of changing the thumb F1 and the index finger F2 from any of the states ST0 to ST5 to the initial state ST0 via the states ST6 and ST7. In other words, the first gesture is an operation of lifting the thumb F1 and lowering the index finger F2. (2) Second gesture The second gesture is the following two types of operations (A) and (B). (A) An operation of changing the thumb F1 and the index finger F2 from the initial state ST0 to any of the states ST1 to ST5. (B) A series of operations of performing the same operation (ST0→ST6→ST7→ST0) within a certain period of time immediately after the operation that is the first gesture (ST0→ST6→ST7→ST0). The operation itself of (B) is the same as the first gesture, but when the operation of (B) is performed immediately after the first gesture and within a certain period of time, it is treated as the second gesture. In other words, the operation of (B) is a double-tap gesture that combines a plurality of operations.

[0109] The first gesture and the second gesture are defined to be different from the normal finger movements of the operator. As a result, the first gesture and the second gesture can be easily detected from among any finger movements. Further, it is preferable that the first gesture is a characteristic finger movement different from the second gesture. Thereby, the control device 30 can easily distinguish between the first gesture and the second gesture. In the present embodiment, the first gesture is a large vertical movement of the thumb, and the second gesture is a movement of the thumb in the front-back, left-right directions in a plane by the index finger or a small downward movement with reference to the same plane.

[0110] FIG. 14 is a diagram showing a combination of peak values of the angular velocity of the thumb F1 used for determining each gesture in the operation input system 2 according to the present embodiment. The circles (“〇”) in the figure indicate the directions in which peak values of the angular velocity (hereinafter referred to as “peak angular velocity”) equal to or greater than a predetermined threshold are detected among the six directions of +X, -X, +Y, -Y, +Z, and -Z. The combination of the directions of the peak angular velocity with circles indicates the characteristics of each gesture. Even when there are no circles, it is assumed that the angular velocity changes slightly.

[0111] First, the change in the angular velocity of the thumb F1 during the second gesture will be described. When performing the second gesture of moving the thumb F1 to the right, the angular velocity of the thumb F1 changes significantly in the three directions of +X, +Y, and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the three directions of +X, +Y, and -Z.

[0112] Similarly, when performing the second gesture of moving the thumb F1 to the left, the angular velocity of the thumb F1 changes significantly in the two directions of -Y and +Z. The second gesture of moving the thumb F1 to the left is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the two directions of -Y and +Z.

[0113] When performing the second gesture of moving the thumb F1 forward, the angular velocity of the thumb F1 changes significantly in five directions: +X, +Y, -Y, +Z, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of five directions: +X, +Y, -Y, +Z, and -Z.

[0114] When performing the second gesture of moving the thumb F1 backward, the angular velocity of the thumb F1 changes significantly in five directions: +X, -X, +Y, -Y, and -Z. The second gesture of moving the thumb F1 backward is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of five directions: +X, -X, +Y, -Y, and -Z.

[0115] When performing the second gesture of tapping the thumb F1, the angular velocity of the thumb F1 changes significantly in three directions: +X, -X, and +Y. The second gesture of tapping the thumb F1 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of three directions: +X, -X, and +Y.

[0116] When performing the second gesture of pressing down the thumb F1, the angular velocity of the thumb F1 changes significantly in two directions: -X and -Z. The second gesture of pressing down the thumb F1 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions: -X and -Z.

[0117] Subsequently, the change in the angular velocity of the thumb F1 during the first gesture will be described. As described above, the first gesture is a series of operations that return the thumb F1 and the index finger F2 from any of the states ST0 to ST5 to the initial state ST0 via the states ST6 and ST7. Therefore, in FIG. 14, the presence or absence of the detection of the peak angular velocity of the thumb F1 during the operation of setting to the release state ST6 and the operation of setting to the initial state ST0 are separately shown.

[0118] During the operation of setting the thumb F1 and the index finger F2 to the release state ST6, the angular velocity of the thumb F1 changes significantly in four directions: +X, +Y, +Z, and -Z. Therefore, the operation of setting to the release state ST6 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of four directions: +X, +Y, +Z, and -Z.

[0119] Also, when the thumb F1 and the index finger F2 are moved from the release state ST6 to the initial state ST0, the angular velocity of the thumb F1 changes significantly in two directions, +X and +Y. The operation of moving the thumb F1 and the index finger F2 from the release state ST6 to the initial state ST0 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, +X and +Y.

[0120] FIG. 15 is a diagram showing a combination of peak angular velocities of the index finger F2 used in the determination of each gesture in the operation input system 2 according to the present embodiment. The circles (“〇”) in the figure are the same as those in FIG. 14.

[0121] First, the change in the angular velocity of the index finger F2 during the second gesture will be described. During the second gesture of moving the thumb F1 to the right, the angular velocity of the index finger F2 changes significantly in two directions, -X and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, -X and -Z.

[0122] During the second gesture of moving the thumb F1 to the left, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the second gesture of moving the thumb F1 to the left, since the combination of the peak angular velocities of the index finger F2 is not detected, there is no feature used for gesture determination.

[0123] During the second gesture of moving the thumb F1 forward, the angular velocity of the index finger F2 changes significantly in three directions, -X, -Y, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of three directions, -X, -Y, and -Z.

[0124] When performing the second gesture of moving the thumb F1 backward, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of moving the thumb F1 backward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, +X and +Y.

[0125] When performing the second gesture of tapping the thumb F1, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the second gesture of tapping the thumb F1, since the combination of the peak angular velocities of the index finger F2 is not detected, there is no feature used for gesture determination.

[0126] When performing the second gesture of pressing down the thumb F1, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of pressing down the thumb F1 is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, +X and +Y.

[0127] Subsequently, the change in the angular velocity of the index finger F2 during the first gesture will be described. In FIG. 15, similar to FIG. 14, the presence or absence of detection of the peak angular velocity during the operation of changing to the release state ST6 and the operation of changing to the initial state ST0 is separately shown.

[0128] During the operation of changing to the release state ST6, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the first half of the first gesture of changing the thumb F1 and the index finger F2 to the release state ST6, since the combination of the peak angular velocities of the index finger F2 is not detected, there is no feature used for gesture determination.

[0129] When operating to change from the release state ST6 to the initial state ST0, the angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the latter half of the first gesture to change from the release state ST6 to the initial state ST0, since the combination of the peak angular velocities of the index finger F2 is not detected, there are no features used for gesture determination.

[0130] FIG. 16 is a diagram for explaining the relationship between a plurality of gestures detected in the operation input system 2 according to the present embodiment. In FIG. 16, eight types of gestures G0 to G6, G8 corresponding to the above-described states ST0 to ST6, ST8 are shown. The direction of the arrow in the figure indicates the direction of transition from one gesture to another gesture. FIG. 17 is a side view for explaining the gesture G5. FIG. 18 is a side view for explaining the gesture G6.

[0131] The gesture G0 is a first gesture in which, starting from the state ST6 where the thumb F1 and the index finger F2 are separated, the thumb F1 is brought into contact with the side surface of the middle phalanx of the index finger F2 to form the state ST7, and then maintained for a certain period of time to reach the initial state ST0.

[0132] The gesture G1 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, while maintaining the contact between the thumb F1 and the index finger F2, the tip of the thumb F1 is moved leftward along the side surface of the index finger F2. The states of the thumb F1 and the index finger F2 are changed from the initial state ST0 to the state ST1 by the gesture G1.

[0133] The gesture G2 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, while keeping the thumb F1 and the index finger F2 in contact, the tip of the thumb F1 is moved rightward along the side surface of the index finger F2 and maintained for a certain period of time. The states of the thumb F1 and the index finger F2 are changed from the initial state ST0 to the state ST2 by the gesture G2.

[0134] Gesture G3 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, while bringing the thumb F1 and the index finger F2 into contact, the tip of the thumb F1 protrudes in an outer direction from the back of the index finger F2. The states of the thumb F1 and the index finger F2 are changed from the initial state ST0 to the state ST3 by the gesture G3.

[0135] Gesture G4 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, while bringing the thumb F1 and the index finger F2 into contact, they are contracted and the tip of the thumb F1 is moved to the side surface of the index finger F2. The states of the thumb F1 and the index finger F2 are changed from the initial state ST0 to the state ST4 by the gesture G4.

[0136] Gesture G5 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, while bringing the thumb F1 and the index finger F2 into contact, the tip of the thumb F1 is pushed down and brought into contact with the index finger F2 and the middle finger F3 as shown in FIG. 17. The states of the thumb F1 and the index finger F2 are changed from the initial state ST0 to the state ST5 by the gesture G5.

[0137] Gesture G6 is a second gesture in which, as shown in FIG. 18, the thumb F1 and the index finger F2 are separated from an arbitrary state and maintained in that state for a certain period of time. For example, the states of the thumb F1 and the index finger F2 are changed from the initial state ST0 or any of the states ST1 to ST5 to the release state ST6 by the gesture G6.

[0138] Gesture G8 is a second gesture in which, when the thumb F1 and the index finger F2 are in the initial state ST0, after the thumb F1 and the index finger F2 are once separated, the side surface of the index finger F2 is tapped a predetermined number of times (for example, 2 times) with the thumb F1 within a predetermined limit time.

[0139] Gestures G1 to G5 and G8 are the second gestures and are pre-associated with six types of commands related to operation inputs. Gestures G1 to G4 correspond to commands for moving a cursor or the like leftward, rightward, upward, and downward on the operation input screen, respectively. Gesture G5 corresponds to a command for canceling the operation input performed immediately before on the operation input screen. Gesture G8 corresponds to a command for determining the selected item or the input information on the operation input screen.

[0140] FIG. 19 is a flowchart showing an example of the processing executed in the operation input system 2 according to the present embodiment. The processing in FIG. 19 is common to the flowchart in FIG. 10 for steps S101 to S108 and S110 to S113. Therefore, the steps different from those in FIG. 10 will be described in detail below.

[0141] In step S108, the control device 30 determines the relative finger movement by two fingers based on the combination of the peak values of the angular velocities extracted during the extraction period. The determination of the finger movement is started when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 is equal to or greater than a predetermined threshold value (step S106: YES), so that the calculation load is suppressed.

[0142] After step S108, the processing proceeds to step S201. In step S201, the control device 30 determines whether the finger movement by two fingers corresponds to the first gesture.

[0143] Here, when the control device 30 determines that the finger movement corresponds to the first gesture, that is, the finger movement is gesture G0 (step S201: YES), the control device 30 updates the detection flag of the first gesture stored in the storage area to ON (step S202), and the processing returns to step S101.

[0144] The detection flag is a flag indicating whether the first gesture has been detected. The detection flag can be initialized to OFF, for example, when a command for operation input is issued. Note that the method of managing the detection state of the first gesture in the operation input system 1 is not limited to the method using the detection flag.

[0145] On the other hand, when the control device 30 determines that the finger movement does not correspond to the first gesture (step S201: NO), the process proceeds to step S203.

[0146] In step S203, the control device 30 determines whether the finger movement is the second gesture. Here, when the control device 30 determines that the finger movement is the second gesture (step S203: YES), the process proceeds to step S204.

[0147] On the other hand, when the control device 30 determines that the finger movement is not the second gesture (step S203: NO), the process returns to step S101.

[0148] In step S204, the control device 30 determines whether the detection flag of the first gesture is ON. That is, the control device 30 determines whether the first gesture has been detected before the second gesture. Here, when the control device 30 determines that the detection flag of the first gesture is ON (step S204: YES), the process proceeds to step S205.

[0149] On the other hand, when the control device 30 determines that the detection flag of the first gesture is not ON (step S204: NO), the process returns to step S101. That is, when the control device 30 determines that the second gesture has been performed before the first gesture is detected, the operation input by the second gesture is not accepted.

[0150] In step S205, the control device 30 outputs a command corresponding to the second gesture. Specifically, when the control device 30 determines that the second gesture has been performed following the first gesture, it accepts the second gesture as an operation input and outputs a command corresponding to the second gesture. Thereafter, the process proceeds to step S110. The processing from step S110 to step S113 is the same as in the case of FIG. 10.

[0151] In the case of FIG. 7 described above, the finger movement from the start time t1 of the extraction period A to the end time t4 of the extraction period B corresponds to the first gesture. Also, the finger movement in the extraction period C following the first gesture corresponds to the second gesture. In the first gesture, a pair of large amplitudes on the plus side and then the minus side occurs for the angular velocity X_1 around the X-axis of the thumb F1, and it is easy to determine the upward finger movement of the thumb F1 in the extraction period A and the downward finger movement of the thumb F1 in the extraction period B. Also, it can be easily determined that the combination of the finger movement in the extraction period A and the finger movement in the extraction period B is the first gesture.

[0152] For example, in FIG. 7, a threshold value THp is set on the plus side and a threshold value THm is set on the minus side for the angular velocity X_1 around the X-axis of the thumb F1. Here, the control device 30 (gesture detection unit 30H) classifies the period in which X_1≥THp as "1", the period in which THm<X_1<THp as "0", and the period in which X_1≤THm as "-1" for the angular velocity X_1. In this case, the control device 30 can easily detect the angular velocity change pattern of "0"→"1"→"0"→"-1"→"0" in the period from time t1 to t6. Then, the control device 30 performs the detection process of the angular velocity change pattern for the other angular velocities Y_1, Z_1, X_2, Y_2, Z_2 in the same manner as the angular velocity X_1. Thereby, the control device 30 can detect the second gesture performed following the first gesture with a small computational load.

[0153] FIG. 20 is a diagram showing an example of an operation input screen displayed on the display device 306 according to the present embodiment. In FIG. 20, a plurality of menus are displayed on the operation input screen, and a state in which menu M1 is selected from among them is shown. Further, in the upper right region of the operation input screen, a cross key K for moving the position of the cursor is displayed. A plurality of indicators are provided around the cross key K. When a command to move the position of the cursor in any of the up, down, left, or right directions is issued by the finger operation of the operator, the indicator in the direction corresponding to the command lights up. In FIG. 20, the indicator below the cross key K is lit in accordance with the issuance of a command to move the position of the cursor downward.

[0154] In a conventional operation input system based on finger gestures, there were cases where the boundary between one gesture and the next could not be detected well. For this reason, there was a possibility of misdetection, where two gestures were regarded as one gesture, or the finger movements of the operator in normal times were regarded as some gesture.

[0155] For example, in a gesture of moving a finger horizontally, if the finger almost stops halfway and then moves further in the same direction, there may be cases where the horizontal movement gesture is detected once, or cases where it is determined that it is repeated twice continuously. Also, if the operator notices a mistake during the movement and moves in the opposite direction, either only one of them may become valid, or both may become valid. Thus, there was a possibility that the gesture made by the operator was detected as a gesture different from the operator's intention.

[0156] In addition, in a conventional operation input system, there was a constraint that finger gestures could not be detected normally unless the palm or the arm was in a predetermined posture (for example, a posture in which the palm was parallel to the ground). When trying to address these problems, very complex calculations were required using multiple types of sensors.

[0157] On the other hand, in the operation input system 2 according to the present embodiment, the operator first performs a first gesture using the thumb F1 and the index finger F2, and then performs a second gesture following the first gesture. When the control device 30 detects a combination of the first gesture and the subsequent second gesture, it outputs a command pre-associated with the second gesture.

[0158] Therefore, the operation input system 2 according to the present embodiment has the advantages of high accuracy in gesture detection and determination and low computational load. Further, since the operation input system 2 is configured to detect the second gesture based on the posture during the first gesture, the operator can input a gesture in any static posture, and the degree of freedom during operation is increased.

[0159] [Third Embodiment] Hereinafter, the operation input system 3 according to the third embodiment will be described. Hereinafter, mainly the differences from the first embodiment and the second embodiment will be described, and the description of the common parts will be omitted or simplified.

[0160] FIG. 21 is a block diagram showing an example of the hardware configuration of the device constituting the operation input system 3 according to the present embodiment. As shown in FIG. 21, the smart glass 40 includes an MCU 401, a wireless communication device 402, a battery 403, and a display device 404. The operation input system 3 according to the present embodiment is different from the first embodiment and the second embodiment in that it further includes the smart glass 40.

[0161] The smart glass 40 is a wearable display that can be worn by the operator as glasses. The smart glass 40 has, for example, an AR (Augmented Reality) function, a camera function, a Bluetooth / Wi-Fi connection function, a microphone function, and the like.

[0162] FIG. 22 is a perspective view showing an example of the smart glasses 40 according to the present embodiment. Here, it is shown that the smart glasses 40 include a spectacle frame 41 and a display device (display unit) 404 that projects a screen onto a lens 42 fitted into the spectacle frame 41. An operation input screen output from the control device 30 as shown in FIG. 20 is projected onto the display device 404.

[0163] The smart glasses 40 according to the present embodiment are controlled by the control device 30 in, for example, the following manner. In a mode where only image / video display is performed, the control device 30 disables operation input by gesture by the OS or an application. Further, when the control device 30 switches the execution mode to a mode that requires input by the operator, the control device 30 enables operation input by gesture by the OS or an application.

[0164] Thereby, a combination of a first gesture performed thereafter and a subsequent second gesture is detected, and operation input by gesture becomes possible. For example, operation input by the operator is performed according to the operation input screen shown in FIG. 20. Thereafter, when the execution mode is switched to a mode where only image / video display is performed, operation input by gesture is disabled again by the OS or an application.

[0165] According to the operation input system 3 of the present embodiment, the operator wears a ring-shaped device on the thumb F1 and the index finger F2 and performs operation input by making a predetermined gesture. An operation input screen is displayed on the smart glasses 40. Then, the operation input screen is updated in real time according to operation input by the operator's finger gesture.

[0166] For example, when the control device 30 is a smartphone, the operator can perform a desired operation input while looking at the screen displayed on the smart glasses 40 instead of the screen displayed on the smartphone. Therefore, the operator can confirm the content of the operation input performed based on the gesture by the first sensor device 10 and the second sensor device 20 by the smart glasses 40, and efficient operation input becomes possible.

[0167] [Modified Embodiment] The present invention is not limited to the above-described embodiments, and can be appropriately modified without departing from the spirit of the present invention.

[0168] For example, in addition to the gyro sensor, the first sensor device 10 and the second sensor device 20 may have an inertial measurement unit further including sensors such as an acceleration sensor and a magnetic sensor as a sensor module. Also, the first sensor device 10 and the second sensor device 20 may use other sensors such as an acceleration sensor instead of the gyro sensor as the sensor module.

[0169] Further, instead of the peak value of the angular velocity during the extraction period, the integrated value of the angular velocity over a predetermined period may be used. By identifying the change pattern of the integrated value, it is possible to specify what kind of finger movement has been performed. This is the same in the case where the first sensor device 10 and the second sensor device 20 are configured to detect acceleration instead of angular velocity.

[0170] Also, the setting information in the first sensor device 10 and the second sensor device 20 is set according to the finger to be worn. Therefore, it is preferable that characters or figures indicating the finger to be worn and the wearing direction are represented on the outer periphery of each base body 11 of the first sensor device 10 and the second sensor device 20. Thereby, the operator can wear the first sensor device 10 and the second sensor device 20 in an optimal state for each finger by referring to the characters or figures on the outer periphery of the first sensor device 10 and the second sensor device 20, so that the accuracy of detecting finger movements can be further improved.

[0171] Also, in the above-described embodiments, the finger movement has been determined based on the relationship between the first angular velocity and the second angular velocity detected by the first sensor device 10 and the second sensor device 20 and a predetermined threshold value. However, the method for determining the finger movement is not limited to this. For example, the control device may be configured to determine the finger movement by inputting the detected first signal and second signal into a learning model that has learned the finger movement corresponding to the first angular velocity (first signal) and the second angular velocity (second signal). When using a learning model for determining the finger movement, it has the effect of further suppressing the computational load.

[0172] In the above-described embodiments, it has been described on the premise that the hand posture is substantially maintained during the period from when the operator performs the first gesture to when the second gesture is completed. However, when the hand posture of the operator is significantly different between the first gesture and the second gesture, the accuracy of gesture detection may decrease or the computational load may increase. Therefore, it is preferable that the control device 30 further includes a configuration for estimating the hand posture other than the first finger and the second finger of the operator based on the first angular velocity (first signal) and the second angular velocity (second signal). Specifically, it estimates the direction of the palm or the back of the hand. By accurately estimating the hand posture, the accuracy of detecting the first gesture and the second gesture can be improved.

[0173] Also, the control device 30 may determine the second gesture on the condition that the change in the hand posture after the detection of the first gesture is within a predetermined range. Thereby, a decrease in the accuracy of gesture detection can be suppressed. Also, it has the effect of reducing the computational load in the control device 30. Further, it is preferable that the control device 30 can feedback to the operator to that effect when it determines that the hand posture at the estimated second gesture is significantly different from the hand posture at the first gesture.

[0174] In the above-described embodiment, the first gesture and the second gesture were detected based on the peak value of the angular velocity during the extraction period and a common threshold value. However, the threshold value used for detecting the first gesture may be set to a value different from the threshold value used for detecting the second gesture. For example, the absolute value of the threshold value when detecting the second gesture may be made smaller than the absolute value of the threshold value when detecting the first gesture. In this case, the absolute values of the angular velocities of the thumb F1 and the index finger F2 in the second gesture become smaller than the absolute values of the angular velocities of the respective fingers in the first gesture. Also, since the sensitivity at the time of detecting the second gesture is higher than that at the time of detecting the first gesture, the operator can perform the second gesture with a smaller movement than in the first gesture after detecting the first gesture.

Explanation of Signs

[0175] 1, 2, 3... operation input system 10... first sensor device 20... second sensor device 30... control device 30A... calibration unit 30B... angular velocity calculation unit 30C... peak value extraction unit 30D... command issuance unit 30E... feedback control unit 30F... display control unit 30G... input unit 30H... gesture detection unit 40... smart glasses 101... MCU 102... sensor module 103... wireless communication device 104... feedback module 105A... tightening mechanism 105B... vibration mechanism 105... battery 201... MCU 202... sensor module 203... wireless communication device 204 ··· Feedback module 205 ··· Battery 301 ··· Processor 302 ··· RAM 303 ··· ROM 304 ··· Storage 305 ··· Communication I / F 306 ··· Display device 307 ··· Input device 401 ··· MCU 402 ··· Wireless communication device 403 ··· Display device 404 ··· Battery

Claims

1. A first sensor device having an annular base body that can be attached to any first finger among the five fingers of one hand of an operator, and outputting a first signal corresponding to the movement of the first finger; A second sensor device having an annular base body that can be attached to a second finger different from the first finger among the five fingers, and outputting a second signal corresponding to the movement of the second finger; A control device for determining a relative finger movement between the first finger and the second finger based on the first signal and the second signal; Comprising: Each of the first sensor device and the second sensor device includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor; Each of the first sensor device and the second sensor device includes a plurality of sensor modules; Each of the first sensor device and the second sensor device includes a feedback module for performing a notification operation to the operator; The feedback module is an operation input system having a tightening mechanism for tightening the first finger or the second finger.

2. The first sensor device includes a first wireless communication device that transmits the first signal to the control device by wireless communication; The second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication, The operation input system according to claim 1.

3. The plurality of sensor modules are arranged at substantially equal intervals on the circumference of the base body, The operation input system according to claim 1.

4. On the outer periphery of the base body, characters or figures indicating the finger to be attached and the attachment direction are represented, The operation input system according to claim 1.

5. The control device determines the attachment states of the first sensor device and the second sensor device based on the first signal and the second signal, The operation input system according to claim 1.

6. The control device performs calibration of the first signal and the second signal when the first sensor device and the second sensor device are activated, The operation input system according to claim 1.

7. The tightening mechanism has an airbag that can expand or contract, The operation input system according to claim 1.

8. The tightening mechanism has a dielectric member that can expand or contract, The operation input system according to claim 1.

9. Each of the first sensor device and the second sensor device includes a plurality of feedback modules corresponding to the plurality of sensor modules respectively, The operation input system according to claim 1.

10. A first sensor device that is attached to any first finger among the five fingers of one hand of an operator and detects a first signal corresponding to the movement of the first finger; A second sensor device that is attached to a second finger different from the first finger among the five fingers and detects a second signal corresponding to the movement of the second finger; A control device that controls the first sensor device and the second sensor device respectively; Comprising: Each of the first sensor device and the second sensor device includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor; Each of the first sensor device and the second sensor device includes a plurality of sensor modules; Each of the first sensor device and the second sensor device includes a feedback module that performs a notification operation to the operator; The feedback module has a clamping mechanism for clamping the first finger or the second finger; The control device: Detects the relative finger movements of the first finger and the second finger based on the first signal and the second signal; Determines whether the finger movement following the first gesture is a predetermined second gesture on the condition that it is determined that the finger movement is a predetermined first gesture; When it is determined that the finger movement is the second gesture, outputs a command corresponding to the second gesture; An operation input system.

11. The control device starts the determination of the finger movement when the first signal or the second signal exceeds a threshold value; The operation input system according to claim 10.

12. The threshold value in the determination of the first gesture is larger than the threshold value in the determination of the second gesture; The operation input system according to claim 11.

13. The control device estimates the posture of the hand of the operator other than the first finger and the second finger based on the first signal and the second signal; The operation input system according to claim 10.

14. The control device determines the second gesture on the condition that the change in the hand posture after the detection of the first gesture is within a predetermined range; The operation input system according to claim 13.

15. The control device determines the finger movement by inputting the detected first signal and second signal into a learning model that has learned the finger movements corresponding to the first signal and the second signal. The operation input system according to claim 10.

16. The first finger is the thumb, and the second finger is the index finger. The first gesture is an operation of separating the thumb and the index finger and then bringing the thumb into contact with the side surface of the index finger. The operation input system according to claim 10.

17. The second gesture is an operation of moving the thumb along the side surface of the index finger in the left, right, forward, and backward directions while keeping the thumb and the index finger in contact with each other. The operation input system according to claim 16.

18. The second gesture is an operation of pressing down the thumb while keeping the thumb and the index finger in contact with each other so that the thumb comes into contact with the index finger and the middle finger. The operation input system according to claim 16.

19. The second gesture is an operation of separating the in-contact thumb and index finger. The operation input system according to claim 16.

20. The second gesture is an operation of bringing the thumb into contact with the index finger after continuously performing upward and downward swings of the thumb a predetermined number of times. The operation input system according to claim 16.

21. The control device outputs a command to move the position of a pointer or a cursor displayed on the operation input screen in any one of the left, right, upward, and downward directions corresponding to the second gesture. The operation input system according to claim 16.

22. The control device outputs a command to cancel the operation input performed immediately before on the operation input screen corresponding to the second gesture. The operation input system according to claim 16.

23. The control device outputs a command to reset a flag indicating the detection state of the first gesture corresponding to the second gesture. The operation input system according to claim 16.

24. The control device outputs a command to determine the operation input performed immediately before on the operation input screen corresponding to the second gesture. The operation input system according to claim 16.

25. The first sensor device includes a first wireless communication device that transmits the first signal to the control device by wireless communication. The second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication. The operation input system according to claim 16.

26. A wearable display including a spectacle frame and a display unit that projects an operation input screen output from the control device onto a lens fitted into the spectacle frame. The operation input system according to claim 16, further comprising the above.

Images