Operation input system

Abstract

To provide an operation input system allowing an operator to perform input operation accurately by a finger.SOLUTION: An operation input system according to the present invention includes: a first sensor device which has an annular base body being wearable to a first finger of an operator, and outputs a first signal according to a motion of the first finger; a second sensor device which has an annular base body being wearable to a second finger different from the first finger and outputs a second signal according to a motion of the second finger; and a control device which determines a relative finger motion of the first finger and the second finger on the basis of the first signal and the second signal.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an operation input system.

Background Art

[0002] Patent Document 1 discloses a smart interface ring that has a pressure-sensitive touch pointer and a three-axis gyro sensor and is worn on one finger of an operator. This smart interface ring captures the movement of the fingertip based on the angular velocity of the finger detected by the three-axis gyro sensor, enabling operation input to an application.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the ring described in Patent Document 1, since it is configured to detect only the angular velocity of one finger of a hand in a predetermined posture state and determine the movement, there is a problem that the finger movement of the operator cannot be accurately determined and the accuracy of the input operation is lowered.

[0005] Therefore, in view of the problems in the above-described conventional technology, an object of the present invention is to provide an operation input system that enables an operator to perform an input operation by a finger movement with high accuracy.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an operation input system is provided, comprising: a first sensor device having an annular base that can be attached to the operator's first finger and outputting a first signal corresponding to the movement of the first finger; a second sensor device having an annular base that can be attached to a second finger different from the first finger and outputting a second signal corresponding to the movement of the second finger; and a control device that determines the relative finger movements of the first finger and the second finger based on the first signal and the second signal.

[0007] According to another aspect of the present invention, an operation input system is provided comprising: a first sensor device attached to the first finger of an operator and detecting a first signal corresponding to the movement of the first finger; a second sensor device attached to a second finger different from the first finger and detecting a second signal corresponding to the movement of the second finger; and a control device that controls the first sensor device and the second sensor device, respectively, wherein the control device detects the relative finger movements of the first finger and the second finger based on the first signal and the second signal, and, on the condition that it determines that the finger movement is a predetermined first gesture, determines whether the finger movement performed following the first gesture is a predetermined second gesture, and if it determines that the finger movement is the second gesture, outputs a command corresponding to the second gesture. [Effects of the Invention]

[0008] According to the present invention, an operation input system is provided that allows an operator to perform input operations by finger movements with high precision. [Brief explanation of the drawing]

[0009] [Figure 1] This block diagram shows an example of the hardware configuration of the device constituting the operation input system according to the first embodiment. [Figure 2] This figure shows an example of how the first sensor device and the second sensor device according to the first embodiment are attached to the operator's hand. [Figure 3] This is a cross-sectional view showing an example of the internal structure of the first sensor device according to the first embodiment. [Figure 4]This is a cross-sectional view showing an example of the internal structure of the first sensor device according to the first embodiment. [Figure 5] This is a functional block diagram showing an example of the functions of the control device according to the first embodiment. [Figure 6] This figure illustrates the time change of the angular velocity of the thumb detected in the first sensor device according to the first embodiment, and the extraction period for the peak value. [Figure 7] This figure shows the time change of the angular velocity of the thumb detected in the first sensor device according to the first embodiment. [Figure 8] This figure shows the time change of the angular velocity of the index finger detected by the second sensor device according to the first embodiment. [Figure 9] This is a timing chart showing the processes and execution times performed in the control device according to the first embodiment. [Figure 10] This flowchart shows an example of a process performed in the operation input system according to the first embodiment. [Figure 11] This is a block diagram showing an example of the overall configuration of the operation input system according to the second embodiment. [Figure 12] This is a functional block diagram showing an example of the functions of the control device according to the second embodiment. [Figure 13] This figure shows the relationship between finger movements and finger state transitions in the operation input system according to the second embodiment. [Figure 14] This figure shows the combinations of peak values ​​of thumb angular velocity used for determining each gesture in the operation input system according to the second embodiment. [Figure 15] This figure shows the combinations of peak values ​​of the angular velocity of the index finger used to determine each gesture in the operation input system according to the second embodiment. [Figure 16] This is a top view illustrating a plurality of gestures detected in the operation input system according to the second embodiment. [Figure 17] This is a side view illustrating the gesture shown in Figure 16. [Figure 18] This is a side view illustrating the gesture shown in Figure 16. [Figure 19] A flowchart showing an example of the processing executed in the operation input system according to the second embodiment. [Figure 20] A diagram showing an example of a screen displayed on the control device according to the second embodiment. [Figure 21] A block diagram showing an example of the hardware configuration of the device constituting the operation input system according to the third embodiment. [Figure 22] A perspective view showing an example of the smart glasses according to the third embodiment.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. Elements having the same functions throughout the drawings are denoted by the same reference numerals, and redundant descriptions may be omitted or simplified.

[0011] [First Embodiment] FIG. 1 is a block diagram showing an example of the hardware configuration of the device constituting the operation input system 1 according to the present embodiment.

[0012] As shown in FIG. 1, the operation input system 1 includes a first sensor device 10, a second sensor device 20, and a control device 30. In the present embodiment, the first sensor device 10 and the second sensor device 20 are connected to the control device 30 via a network.

[0013] The first sensor device 10 has an annular base body that can be worn on the first finger of the operator, and outputs a first signal corresponding to the movement of the first finger. The first sensor device 10 of the present embodiment outputs detection data of the angular velocity of the first finger (hereinafter referred to as "first angular velocity") as a first signal corresponding to the movement of the first finger. Specifically, the first sensor device 10 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

[0014] As shown in Figure 1, the first sensor device 10 comprises an MCU (Micro Controller Unit) 101, a sensor module 102, a feedback module 103, a wireless communication device 104, and a battery 105. The various parts of the first sensor device 10 are connected by wiring on a circuit board (not shown).

[0015] The MCU101 is an embedded microprocessor that integrates a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), I / O devices, and other components into a single integrated circuit. The MCU101 is primarily used for controlling electronic devices.

[0016] The sensor module 102 has a gyro sensor for detecting angular velocity. The gyro sensor in this embodiment is a three-axis gyro sensor that detects angular velocity around the mutually orthogonal X, Y, and Z axes. In addition to the gyro sensor, the sensor module 102 may also be equipped with other sensors such as an accelerometer and a magnetic sensor. When using an accelerometer, the actual acceleration is calculated by subtracting the component of gravity from the detected acceleration value.

[0017] The sensor module 102 applies a voltage to the MCU 101 that is proportional to the detected angular velocity. The MCU 101 performs A / D conversion of the voltage applied from the sensor module 102 into digital data. The MCU 101 then transmits the detected angular velocity data to the control device 30 via the wireless communication device 104, which will be described later.

[0018] The feedback module 103 notifies the operator wearing the first sensor device 10 on their thumb based on the control information received from the control device 30. Details of the feedback module 103 will be described later.

[0019] The wireless communication device 104 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for wireless communication with other devices. The wireless communication device 104 transmits the detection data of the first angular velocity to the control device 30 via wireless communication.

[0020] The battery 105 is a device that supplies power for driving each of the MCU 101, sensor module 102, feedback module 103, and wireless communication device 104.

[0021] The second sensor device 20 has an annular base that can be attached to a second finger different from the operator's first finger, and outputs a second signal corresponding to the movement of the second finger. In this embodiment, the second sensor device 20 outputs detection data of the angular velocity of the second finger (hereinafter referred to as "second angular velocity") as a second signal corresponding to the movement of the second finger. Specifically, the second sensor device 20 transmits the detection data of the second angular velocity to the control device 30 via wireless communication.

[0022] In this embodiment, the first finger is the thumb. The second finger is preferably the index finger, as it is the finger adjacent to the first finger. Furthermore, the setting information in the first sensor device 10 and the second sensor device 20 is set to match the thumb and index finger of the device to be worn.

[0023] As shown in Figure 1, the second sensor device 20 comprises an MCU 201, a sensor module 202, a feedback module 203, a wireless communication device 204, and a battery 205. Since the hardware configuration of the second sensor device 20 is the same as that of the first sensor device 10, a description of each device is omitted.

[0024] The control device 30 is a computer device that performs calculations, control, and storage. Examples of the control device 30 include personal computers, laptop computers, tablet terminals, and smartphones. The control device 30 determines the relative finger movements of the first and second fingers based on the first and second angular velocities.

[0025] As shown in Figure 1, the control device 30 includes a processor 301, RAM 302, ROM 303, storage 304, communication interface 305, display device 306, and input device 307. Each device is interconnected via a bus, wiring, drive devices, etc.

[0026] The processor 301 performs predetermined calculations according to programs stored in the ROM 303, storage 304, etc., and has the function of controlling each part of the control device 30, the first sensor device 10, and the second sensor device 20. The processor 301 can be a CPU, GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), etc.

[0027] RAM 302 is a memory device composed of a volatile storage medium that provides a temporary memory area necessary for the operation of the processor 301. RAM 302 may be, for example, D-RAM (Dynamic RAM).

[0028] ROM 303 is composed of a non-volatile storage medium and stores necessary information such as programs used for the operation of the control device 30. ROM 303 may be, for example, a P-ROM (Programmable ROM).

[0029] The storage device 304 is composed of a non-volatile storage medium and is a storage device that stores processing data in the control device 30 and operating programs for the control device 30. The storage device 304 is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0030] The processor 301 loads programs stored in ROM 303, storage 304, etc., into RAM 302 and executes them.

[0031] The communication I / F305 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for wireless communication with the first sensor device 10 and the second sensor device 20.

[0032] The display device 306 displays videos, still images, text, etc. The display device 306 can be a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or the like. The operation input screen displayed on the display device 306 includes graphics such as windows, icons, and buttons. The operator selects the graphics representing the desired action on the operation input screen using an operation input means with a pointing device, or an operation input means with the first sensor device 10 and the second sensor device 20.

[0033] The input device 307 is a keyboard, pointing device, or the like, used by the operator to operate the control device 30. Examples of pointing devices include a mouse, trackball, touch panel, and pen tablet. The display device 306 and the input device 307 may be integrally formed as a touch panel.

[0034] Note that the hardware configuration shown in Figure 1 is an example, and other devices may be added, or some devices may be omitted. Also, some devices may be replaced with other devices having similar functions. Furthermore, some functions of the first embodiment may be provided by other devices via a network, or the functions of the first embodiment may be implemented by distributing them across multiple devices. The illustrated hardware configuration can be modified as appropriate.

[0035] Figure 2 shows an example of the first sensor device 10 and the second sensor device 20 according to this embodiment being attached to the operator's fingers. In Figure 2, the first sensor device 10 is attached to the base of the proximal phalanx of the operator's thumb F1. The second sensor device 20 is attached to the base of the proximal phalanx of the operator's index finger F2.

[0036] Furthermore, in the XYZ coordinate system of the second sensor device 20, the X axis is set in the width direction of the operator's index finger F2, the Y axis is set in the direction in which the proximal phalanx of the index finger F2 extends, and the Z axis is set in the thickness direction of the index finger F2. The X, Y, and Z axes are orthogonal to each other. Similarly to the second sensor device 20, in the XYZ coordinate system of the first sensor device 10, the X axis is set in the width direction of the operator's thumb F1, the Y axis is set in the direction in which the proximal phalanx of the thumb F1 extends, and the Z axis is set in the thickness direction of the thumb F1.

[0037] Figures 3 and 4 are cross-sectional views showing an example of the internal structure of the first sensor device 10 according to this embodiment. Since the internal structure of the second sensor device 20 is the same as that of the first sensor device 10, the first sensor device 10 will be described below as a representative example.

[0038] Figure 3 shows an example in which a tightening mechanism 103A is used as the feedback module 103 in the first sensor device 10. The tightening mechanism 103A expands or contracts based on control information from the control device 30. Examples of the tightening mechanism 103A include an airbag that can expand or contract in response to an applied voltage, or a dielectric material (e.g., e-Rubber, a next-generation rubber material manufactured by Toyoda Gosei Co., Ltd.). When the tightening mechanism 103A expands, it deforms into a protruding shape toward the operator's finger, applying a pressing force to the operator's finger. As a result, the control device 30 causes the feedback module 103 to perform feedback in response to the operator's input. The feedback is performed to notify the operator that the operator's input operation has been received and that a predetermined command has been executed.

[0039] In Figure 3, the tightening mechanism 103A is formed in a ring shape, similar to the base body 11, but the shape of the tightening mechanism 103A is not limited to this. The tightening mechanism 103A may be formed in a sheet shape, for example. Also, the number of tightening mechanisms 103A is one, but is not limited to this.

[0040] If multiple tightening mechanisms 103A are provided, there will be multiple locations within the device where expansion or contraction is possible. Therefore, by switching the tightening mechanism 103A that is targeted for expansion or contraction within the first sensor device 10 according to time, feedback in clockwise, counterclockwise, etc., becomes possible. In other words, it becomes possible to further increase the variations in the feedback method.

[0041] Furthermore, the first sensor device 10 is equipped with three sensor modules 102. In a cross-sectional view, the three sensor modules 102 are arranged at approximately equal intervals around the base body 11. Sensors such as gyro sensors have different angular velocity values ​​depending on the mounting position. Therefore, by providing the three sensor modules 102 spaced apart from each other, angular velocity can be detected at different positions. As a result, the control device 30 can determine the mounting state of the first sensor device 10 on the thumb F1 with high accuracy based on multiple angular velocities. The same applies to the second sensor device 20.

[0042] On the other hand, Figure 4 shows an example in which three vibration mechanisms 103B are used as the feedback module 103 in the first sensor device 10. The number of sensor modules 102 is three, as in the case of Figure 3.

[0043] The vibration mechanism 103B vibrates based on control information from the control device 30. The vibration mechanism 103B is composed of an electromagnetic drive unit, such as a piezoelectric element (piezo element) that can expand or contract in response to an applied voltage, and a vibrating vibration motor. A piezo element is a passive element that uses the piezoelectric effect and inverse piezoelectric effect that occur in dielectrics such as quartz or silica to control and detect minute movements. Because it has a simple structure that does not require gears or motors for operation, it is a small element compared to other minute movement mechanism elements.

[0044] Furthermore, the vibration mechanism 103B does not have to be installed adjacent to the sensor module 102, but may be installed between multiple sensor modules 102. When the vibration mechanism 103B vibrates, the vibration is transmitted to the operator's finger. As a result, the control device 30 causes the feedback module 103 to perform feedback in response to the operator's input.

[0045] In Figure 4, the number of vibration mechanisms 103B is the same as the number of sensor modules 102, but the number of vibration mechanisms 103B is not limited to this. The magnitude of the vibration and the vibration period in each module may be changed.

[0046] If multiple vibration mechanisms 103B are provided, there will be multiple locations within the device where vibration is possible. By switching the vibration mechanisms 103B that vibrate within the first sensor device 10 according to time, feedback in clockwise, counterclockwise, etc., becomes possible. In other words, it becomes possible to further increase the variations in the feedback method.

[0047] Figure 5 is a functional block diagram showing an example of the functions of the control device 30 according to this embodiment. The control device 30 includes a calibration unit 30A, an angular velocity calculation unit 30B, a peak value extraction unit 30C, a command issuance unit 30D, a feedback control unit 30E, a display control unit 30F, and an input unit 30G.

[0048] The calibration unit 30A performs calibration on the angular velocity detection data detected by the sensor modules 102 and 202 when the first sensor device 10 and the second sensor device 20 are attached to the operator's finger and activated for the first time. The calibration unit 30A outputs the correction value (offset value) of the angular velocity calculated by the calibration to the angular velocity calculation unit 30B.

[0049] The angular velocity calculation unit 30B calculates the angular velocities of the operator's thumb F1 and index finger F2 based on the angular velocity detection data acquired wirelessly from the first sensor device 10 and the second sensor device 20 and the correction value calculated by the calibration unit 30A. The angular velocity calculation unit 30B outputs the calculated angular velocity to the peak value extraction unit 30C. The angular velocity calculation process in the angular velocity calculation unit 30B is repeatedly performed each time angular velocity detection data is received from the first sensor device 10 and the second sensor device 20.

[0050] The peak value extraction unit 30C extracts peak values ​​from the angular velocities within a predetermined extraction period. The peak value extraction unit 30C outputs the extracted peak values ​​to the command issuing unit 30D.

[0051] Figure 6 shows an example of the time variation of the angular velocity detected in the first sensor device 10 according to this embodiment and the extraction period for the peak value. In Figure 6, as shown by the dashed line, the positive threshold THp for angular velocity is set to +20 rad / second, and the negative threshold THm is set to -20 rad / second. The absolute values ​​of the thresholds THp and THm are equal. The extraction period T is defined as the time from time t1, when at least one of the angular velocities around the X, Y, and Z axes of the two fingers exceeds a predetermined threshold, to time t2, when a certain amount of time has elapsed. However, the values ​​of the thresholds THp and THm and the method of setting the extraction period T are not limited to these.

[0052] The command issuing unit 30D determines the relative finger movements of the operator's two fingers based on the peak values ​​extracted within the extraction period and outputs a command that is pre-associated with that finger movement. The command issuing unit 30D is intended to detect at least two types of finger movements. When the command issuing unit 30D outputs a command, processing corresponding to the command is executed in one of the devices constituting the operation input system 1.

[0053] The feedback control unit 30E transmits control information to the MCU 101 of the first sensor device 10 and the MCU 201 of the second sensor device 20, causing the feedback modules 103 and 203 to execute predetermined feedback processing.

[0054] The display control unit 30F controls the display of the operation input screen shown on the display device 306 based on input operations received by the input device 307 at the input unit 30G, and input operations received by the first sensor device 10 and the second sensor device 20.

[0055] The input unit 30G receives information entered by users, such as operators or administrators, using the input device 307 to operate the control device 30.

[0056] Figure 7 shows the time change of the angular velocity of the thumb F1 detected by the first sensor device 10 according to this embodiment. On the other hand, Figure 8 shows the time change of the angular velocity of the index finger F2 detected by the second sensor device 20 according to this embodiment. In Figures 7 and 8, extraction period A is shown from time t1 to t2, extraction period B is shown from time t3 to t4, and extraction period C is shown from time t5 to t6.

[0057] Furthermore, the threshold values ​​THp and THm for the angular velocity of thumb F1 in Figure 7 are different from the threshold values ​​THp and THm for the angular velocity of index finger F2 in Figure 8. Thus, it is preferable that the threshold values ​​THp and THm be set to optimal values ​​for each finger. The control device 30 determines the relative finger movements of thumb F1 and index finger F2 in each of the extraction periods A, B, and C based on the peak values ​​of the angular velocities of the two fingers around the X, Y, and Z axes, and the threshold values ​​THp and THm in each of the extraction periods A, B, and C.

[0058] As shown in Figure 7, during extraction period A, the angular velocity X_1 of thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is greater than the positive threshold THp. Also, the angular velocity Y_1 of thumb F1 around the Y-axis is displaced on the negative side, and the peak value on the negative side is greater than the negative threshold THm. The angular velocity Z_1 of thumb F1 around the Z-axis is displaced on the positive side, and the peak value on the positive side is less than the threshold THp.

[0059] Furthermore, as shown in Figure 8, during extraction period A, the angular velocity X_2 of the index finger F2 around the X-axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold THp. Also, the angular velocity Y_2 of the index finger F2 around the Y-axis is displaced on the positive side, and the peak value on the positive side is greater than the threshold THp. The angular velocity Z_2 of the index finger F2 around the Z-axis is displaced on both the positive and negative sides, with the peak value on the positive side being less than the threshold THp and the peak value on the negative side being greater than the threshold THm.

[0060] Analyzing the changes in angular velocity in Figures 7 and 8 for extraction period A, it can be seen that the thumb F1 moved significantly in the +X direction around the X axis. Also, during extraction period A, it can be seen that the index finger F2 moved significantly in the +Y direction around the Y axis. As a result, a finger movement was detected in which the thumb F1 was swung upwards more than the index finger F2 during extraction period A.

[0061] As shown in Figure 7, during extraction period B, the angular velocity X_1 of thumb F1 around the X-axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold THm. Also, the angular velocity Y_1 of thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold THp. The angular velocity Z_1 of thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold THm.

[0062] Furthermore, as shown in Figure 8, during extraction period B, the angular velocity X_2 of the index finger F2 around the X-axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold THm. Also, the angular velocity Y_2 of the index finger F2 around the Y-axis is displaced on the negative side, and the peak value on the negative side is less than the threshold THm. The angular velocity Z_2 of the index finger F2 around the Z-axis is displaced on the positive side, and the peak value on the positive side is less than the threshold THp.

[0063] Analyzing the changes in angular velocity in Figures 7 and 8 for extraction period B, it can be seen that the thumb F1 moved significantly in the -X direction around the X axis. Also, during extraction period B, it can be seen that the index finger F2 moved significantly in the -Y direction around the Y axis. As a result, a finger movement in which the thumb F1 touches the index finger F2 is detected during extraction period B.

[0064] As shown in Figure 7, during extraction period C, the angular velocity X_1 of thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold THp. Also, the angular velocity Y_1 of thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold THp. The angular velocity Z_1 of thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold THm.

[0065] Furthermore, as shown in Figure 8, during extraction period C, the angular velocity X_2 of the index finger F2 around the X-axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold THm. Also, the angular velocity Y_2 of the index finger F2 around the Y-axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold THp. The angular velocity Z_2 of the index finger F2 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold THm.

[0066] Analyzing the changes in angular velocity in Figures 7 and 8 over extraction period C, it can be seen that the thumb F1 moved significantly in the +Y direction around the Y axis. Also, during extraction period C, it can be seen that the index finger F2 moved significantly in the -X direction around the X axis and in the +Y direction around the Y axis. As a result, during extraction period C, a finger movement was detected in which the thumb F1 slid to the right along the side of the index finger F2 while maintaining contact with the thumb F1.

[0067] Figure 9 is a timing chart showing the processes and execution times performed in the control device 30 according to this embodiment. Here, the correspondence between the extraction periods A, B, and C shown in Figures 7 and 8, and the command issuance process, the feedback destination determination process, the drive control process for the feedback module, and the display control process for the operation input screen is shown.

[0068] The initial command issuance process in the command issuing unit 30D is executed during extraction period A from time t1 to t2. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t2 to t21. Then, the drive control process of the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t21 to t22.

[0069] Furthermore, the second command issuance process in the command issuance unit 30D is executed during extraction period B from time t3 to t4. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t4 to t41. Then, the drive control process of the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t41 to t42.

[0070] Then, the third command issuance process in the command issuance unit 30D is executed during extraction period C from time t5 to t6. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t6 to t61. Then, the drive control process of the feedback modules 103 and 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t61 to t62.

[0071] Figure 10 is a flowchart showing an example of a process performed in the control device 30 according to this embodiment. This process is repeatedly performed, for example, while the operation input screen is displayed.

[0072] First, the control device 30 acquires angular velocity detection data from the first sensor device 10 and the second sensor device 20 via wireless communication (step S101).

[0073] Next, the control device 30 determines whether or not it is the first time the first sensor device 10 and the second sensor device 20 are being started (step S102). If the control device 30 determines that it is the first time the devices are being started (step S102: YES), the process proceeds to step S103. On the other hand, if the control device 30 determines that it is not the first time the devices are being started (step S102: NO), the process proceeds to step S105.

[0074] In step S103, the control device 30 performs calibration based on the angular velocity detection data acquired from the first sensor device 10 and the second sensor device 20, and calculates an angular velocity correction value. After that, the process proceeds to step S104.

[0075] In step S104, the control device 30 determines the mounting status of the sensor based on the angular velocity detection data. The process then proceeds to step S105.

[0076] As shown in Figure 3, the first sensor device 10 is equipped with multiple sensor modules 102. Therefore, the control device 30 can determine the mounting state of the first sensor device 10 on the thumb F1 based on multiple detection data detected by the multiple sensor modules 102.

[0077] Similarly, the second sensor device 20 includes multiple sensor modules 202. The control device 30 can determine the wearing state of the second sensor device 20 on the index finger F2 based on multiple detection data detected by the multiple sensor modules 202.

[0078] In step S105, the control device 30 calculates the angular velocities of the thumb F1 and index finger F2 based on the detected angular velocity data and the angular velocity correction value. The angular velocities of the thumb F1 and index finger F2 are calculated for the three axes: the X axis, Y axis, and Z axis.

[0079] Next, the control device 30 determines whether any of the calculated absolute values ​​of the angular velocities are greater than or equal to a predetermined threshold (step S106).

[0080] If the control device 30 determines that the absolute value of any angular velocity is greater than or equal to a predetermined threshold (step S106: YES), the process proceeds to step S107. Conversely, if the control device 30 determines that the absolute values ​​of all angular velocities are less than a predetermined threshold (step S107: NO), the process returns to step S101.

[0081] In step S107, the control device 30 determines the angular velocity extraction period and extracts peak values ​​from the angular velocities within the extraction period. Peak values ​​are extracted for each of the three axes: the X, Y, and Z axes. If the angular velocity fluctuates within the extraction period, multiple peak values ​​may be extracted in the same direction. In such cases, the control device 30 sequentially updates the peak values ​​stored in the memory area, for example.

[0082] Next, the control device 30 determines the relative finger movement of two fingers based on the combination of peak angular velocity values ​​of the two fingers extracted during the extraction period (step S108). Since the determination of finger movement is initiated when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 exceeds a predetermined threshold, the computational load is suppressed.

[0083] In step S108, for example, finger movements such as the separation of the thumb F1 and index finger F2, the bringing of the thumb F1 and index finger F2 into contact, and the movement of the thumb F1 moving up and down, left and right, or forward and backward along the index finger F2 while the thumb F1 and index finger F2 are in contact are detected.

[0084] Next, the control device 30 determines a command corresponding to the relative movement of two fingers and outputs that command (step S109). The control device 30 then executes the process specified by the command. Examples of commands include commands to move the cursor or pointer position up, down, left, or right on the operation input screen, commands to cancel the most recent operation input, and commands to determine the operation input.

[0085] Next, the control device 30 determines the feedback destination in accordance with the execution of the issued command (step S110). In this embodiment, the first sensor device 10, the second sensor device 20, and the display device 106 can be the feedback destinations.

[0086] Next, the control device 30 transmits control information to the feedback modules 103 and 203, which are the determined feedback destinations, and drives the feedback modules 103 and 203 (step S111).

[0087] Next, the control device 30 updates the operation input screen displayed on the display device 106 (step S112). For example, the control device 30 changes the coordinates of the pointer on the operation input screen, or changes the display state of the selected menu. In this way, the control device 30 notifies the GUI (display device) in synchronization with the notification to the feedback module. After that, the process ends.

[0088] The control device 30 then determines whether the operator has instructed the end of the operation input on the operation input screen (step S113). If the control device 30 determines that the operator has instructed the end of the operation input (step S113: YES), the process ends. On the other hand, if the control device 30 determines that the operator has not instructed the end of the operation input (step S113: NO), the process returns to step S101.

[0089] Traditionally, input methods that replace mouse operation have included gestures, controllers, eye-tracking, and voice input. However, gestures and controllers involve large movements, making the user's actions easily noticeable to others. Eye-tracking input is difficult to use consistently due to fluctuating eye position. Voice input allows others to hear the user giving instructions. In light of these problems, there has been a need for a pointing device that is discreet and easy to use.

[0090] In contrast, according to the operation input system 1 of this embodiment, a first sensor device 10 and a second sensor device 20, which are ring-shaped devices equipped with gyro sensors, are attached to the thumb (first finger) F1 and the index finger (second finger) F2, respectively. The first sensor device 10 and the second sensor device 20 transmit the angular velocity of the two fingers to the control device 30 via wireless communication. The control device 30 uses the angular velocity of the thumb F1 and the index finger F2 to detect the relative finger movement of the two fingers.

[0091] This allows the system to detect and output commands corresponding to finger movements, enabling free and highly accurate operation input. Furthermore, because the system is configured to allow operation input simply by moving the thumb F1 and index finger F2, it is difficult for others to know that the user is performing operation input, which is beneficial from the perspective of improving information security and protecting privacy.

[0092] Furthermore, in conventional systems where a ring-shaped device was attached to only one finger, it was necessary to change the direction of the arm in order to indicate a specific position with the finger. In contrast, the operation input system 1 according to this embodiment is configured to determine the relative movement of two fingers by attaching ring-shaped devices equipped with gyro sensors to the thumb F1 and index finger F2, respectively. Therefore, pointing becomes possible in any arm orientation.

[0093] Furthermore, conventional ring-shaped devices are known to be equipped with gyro sensors for motion detection. However, because the information that can be detected changes depending on the mounting position of the gyro sensor, the ring-shaped device always had to be worn in a specific direction. In contrast, according to the operation input system 1 of this embodiment, the ring-shaped device (first sensor device 10 and second sensor device 20) has multiple sensor modules mounted at different positions within the ring. Since the wearing state of the ring-shaped device can be determined based on multiple detection data from multiple sensor modules, the ring-shaped device can be worn at any position on each finger.

[0094] [Second Embodiment] The following describes the operation input system 2 according to the second embodiment. The following description will primarily focus on the differences from the first embodiment, while common parts will be omitted or simplified.

[0095] Figure 11 is a block diagram showing an example of the hardware configuration of the operation input system 2 according to this embodiment. The hardware configuration of the operation input system 2 is the same as that of the operation input system 1 according to the first embodiment.

[0096] Figure 12 is a functional block diagram showing an example of the functions of the control device 30 according to this embodiment. Unlike the first embodiment, the control device 30 of this embodiment further includes a gesture detection unit 30H.

[0097] The gesture detection unit 30H detects a specific finger movement from among arbitrary finger movements performed by the thumb F1 and index finger F2 based on the angular velocity of the thumb F1 and index finger F2. In this embodiment, the specific finger movement that the gesture detection unit 30H detects from among arbitrary finger movements is called a "gesture". Note that a gesture is not limited to a single finger movement. Multiple finger movements performed continuously within a certain period of time may also be defined as a single gesture. Furthermore, the correspondence between multiple gestures and the commands issued for each gesture is assumed to be defined in advance. The command issuing unit 30D in this embodiment outputs a command corresponding to the specific gesture detected by the gesture detection unit 30H.

[0098] Figure 13 is a diagram illustrating the relationship between finger movements and finger state transitions in the operation input system 2 according to this embodiment. Here, finger movements are shown as "thumb up", "thumb down", "thumb down", "still", and "movement". The finger states are shown as "initial (ST0)", "left (ST1)", "right (ST2)", "forward (ST3)", "backward (ST4)", "down (ST5)", "release (ST6)", and "contact (ST7)".

[0099] The "initial state (ST0)" is when the operator lightly clenches their hand and places their thumb F1 midway between the first and second joints of their index finger F2. Hereafter, the "initial state (ST0)" will be referred to as the initial state ST0.

[0100] "Left (ST1)" is the state in which the thumb F1 and index finger F2, which are in the initial state ST0, are in contact, and the tip of the thumb F1 is moved to the left along the side of the index finger F2. Hereafter, "Left (ST1)" will be referred to as state ST1.

[0101] "Right (ST2)" is the state in which the thumb F1 and index finger F2, which are in contact in the initial state ST0, are moved to the right along the side of the index finger F2. Hereafter, "Right (ST2)" will be referred to as state ST2.

[0102] "Forward (ST3)" is the state in which the thumb F1 and index finger F2 are in contact with each other, as in the initial state ST0, and the tip of the thumb F1 is extended outward from the proximal phalanx of the index finger F2. Hereafter, "forward (ST3)" will be referred to as state ST3.

[0103] "Post-position (ST4)" is the state in which the thumb F1 and index finger F2, which are in contact in the initial state ST0, are moved to the side of the proximal phalanx of the index finger F2. Hereafter, "post-position (ST4)" will be referred to as state ST4.

[0104] "Down (ST5)" is the state in which the thumb F1 and index finger F2 are in contact, as in the initial state ST0, and the tip of the thumb F1 is pressed down to a position below the index finger F2. Hereafter, "Down (ST5)" will be referred to as state ST5. In state ST5, the thumb F1 is positioned between the index finger F2 and the middle finger F3.

[0105] "Release (ST6)" is the state when the thumb F1 and index finger F2, which were in contact, are separated and held in that position for a certain period of time. Hereafter, "Release (ST6)" will be referred to as Release State ST6.

[0106] "Contact (ST7)" is the state in which the thumb F1 and index finger F2, which were in the released state ST6, are in contact. Hereafter, "Contact (ST7)" will be referred to as state ST7. State ST7 is an intermediate state when changing from the released state ST6 to the initial state ST0.

[0107] The control device 30 (gesture detection unit 30H) determines the state of the thumb F1 and index finger F2 based on the change in angular velocity. The control device 30 determines that the current state is the initial state ST0 when the state transitions from the release state ST6 to the contact state ST7 and the contact state ST7 continues for a certain period of time.

[0108] Furthermore, in this embodiment, the first gesture and the second gesture are defined as follows. (1) First gesture The first gesture is the action of moving the thumb F1 and index finger F2 from one of states ST0 to ST5, through states ST6 and ST7, back to the initial state ST0. In other words, the first gesture is the action of raising the thumb F1 and lowering the thumb F2. (2) Second gesture The second gesture consists of the following two types of actions: (A) and (B). (A) The action of changing the thumb F1 and index finger F2 to one of the states ST1 to ST5, starting from the initial state ST0. (B) A series of actions in which the same actions (ST0→ST6→ST7→ST0) are performed within a certain time period immediately following the first gesture (ST0→ST6→ST7→ST0). Action (B) is the same as the first gesture, but if action (B) is performed immediately after the first gesture and within a certain period of time, it is treated as the second gesture. In other words, action (B) is a double-tap gesture that combines multiple actions.

[0109] The first and second gestures are defined to be different from the operator's normal finger movements. This allows for easy detection of the first and second gestures from among any finger movements. Furthermore, it is preferable that the first gesture be a distinctive finger movement different from the second gesture. This allows the control device 30 to easily distinguish between the first and second gestures. In this embodiment, the first gesture is a large up-and-down movement of the thumb, and the second gesture is a forward-backward, left-and-right movement of the thumb in a plane by the index finger, or a small downward movement relative to the same plane.

[0110] Figure 14 shows the combinations of peak angular velocity values ​​of the thumb F1 used for determining each gesture in the operation input system 2 according to this embodiment. The circles ("〇") in the figure indicate the direction in which a peak angular velocity value (hereinafter referred to as "peak angular velocity") above a predetermined threshold is detected, out of the six directions: +X, -X, +Y, -Y, +Z, and -Z. The combinations of peak angular velocity directions indicated by the circles represent the characteristics of each gesture. Note that even if there are no circles, the angular velocity is assumed to be changing slightly.

[0111] First, let's explain the change in the angular velocity of thumb F1 during the second gesture. During the second gesture, when thumb F1 is moved to the right, the angular velocity of thumb F1 changes significantly in three directions: +X, +Y, and -Z. The second gesture, when thumb F1 is moved to the right, is characterized in that the peak angular velocity of thumb F1 is detected in combinations of the three directions: +X, +Y, and -Z.

[0112] Similarly, during the second gesture of moving thumb F1 to the left, the angular velocity of thumb F1 changes significantly in two directions, -Y and +Z. The second gesture of moving thumb F1 to the left is characterized in that the peak angular velocity of thumb F1 is detected in a combination of the two directions, -Y and +Z.

[0113] During the second gesture of moving thumb F1 forward, the angular velocity of thumb F1 changes significantly in five directions: +X, +Y, -Y, +Z, and -Z. The second gesture of moving thumb F1 forward is characterized in that the peak angular velocity of thumb F1 is detected in combinations of the five directions: +X, +Y, -Y, +Z, and -Z.

[0114] During the second gesture of moving thumb F1 backward, the angular velocity of thumb F1 changes significantly in five directions: +X, -X, +Y, -Y, and -Z. The second gesture of moving thumb F1 backward is characterized in that the peak angular velocity of thumb F1 is detected in combinations of the five directions: +X, -X, +Y, -Y, and -Z.

[0115] During the second gesture of tapping thumb F1, the angular velocity of thumb F1 changes significantly in three directions: +X, -X, and +Y. The second gesture of tapping thumb F1 is characterized by the detection of the peak angular velocity of thumb F1 in combinations of the three directions: +X, -X, and +Y.

[0116] During the second gesture of pressing down the thumb F1, the angular velocity of the thumb F1 changes significantly in two directions, -X and -Z. The second gesture of pressing down the thumb F1 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the two directions, -X and -Z.

[0117] Next, we will explain the change in angular velocity of the thumb F1 during the first gesture. As mentioned above, the first gesture is a series of actions that move the thumb F1 and index finger F2 from one of states ST0 to ST5, through states ST6 and ST7, to the initial state ST0. For this reason, Figure 14 shows separately whether or not the peak angular velocity of the thumb F1 is detected during the action to move to the release state ST6 and the action to move to the initial state ST0.

[0118] During the release of the thumb F1 and index finger F2 to the ST6 position, the angular velocity of the thumb F1 changes significantly in the four directions: +X, +Y, +Z, and -Z. Therefore, the release of the thumb to the ST6 position is characterized by the detection of the peak angular velocity of the thumb F1 in combinations of the four directions: +X, +Y, +Z, and -Z.

[0119] Furthermore, during the operation of moving the thumb F1 and index finger F2 from the released state ST6 to the initial state ST0, the angular velocity of the thumb F1 changes significantly in two directions, +X and +Y. The operation of moving the thumb F1 and index finger F2 from the released state ST6 to the initial state ST0 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the two directions, +X and +Y.

[0120] Figure 15 shows the combinations of peak angular velocities of the index finger F2 used for determining each gesture in the operation input system 2 according to this embodiment. The circles ("〇") in the figure are the same as in Figure 14.

[0121] First, let's explain the change in angular velocity of the index finger F2 during the second gesture. During the second gesture, when the thumb F1 is moved to the right, the angular velocity of the index finger F2 changes significantly in two directions, -X and -Z. The second gesture, when the thumb F1 is moved to the right, is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the two directions, -X and -Z.

[0122] During the second gesture, where the thumb F1 is moved to the left, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in all directions. Since no combination of peak angular velocities of the index finger F2 is detected during the second gesture, where the thumb F1 is moved to the left, there are no features to use for gesture detection.

[0123] During the second gesture of moving the thumb F1 forward, the angular velocity of the index finger F2 changes significantly in the three directions -X, -Y, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the index finger F2 is detected in combinations of the three directions -X, -Y, and -Z.

[0124] During the second gesture of moving the thumb F1 backward, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of moving the thumb F1 backward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the two directions, +X and +Y.

[0125] During the second gesture of tapping thumb F1, the peak angular velocity of index finger F2 is not detected in any direction. However, the angular velocity of index finger F2 changes slightly in all directions. Since no combination of peak angular velocities of index finger F2 is detected during the second gesture of tapping thumb F1, there are no features to use for gesture detection.

[0126] During the second gesture of pressing down the thumb F1, the angular velocity of the index finger F2 changes significantly in two directions, +X and +Y. The second gesture of pressing down the thumb F1 is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the two directions, +X and +Y.

[0127] Next, we will explain the change in angular velocity of the index finger F2 during the first gesture. In Figure 15, similar to Figure 14, we show the presence or absence of peak angular velocity detection during the release state ST6 and the initial state ST0 separately.

[0128] During the action of bringing the thumb to the release state ST6, the peak angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in all directions. In the first half of the first gesture, which brings the thumb F1 and index finger F2 to the release state ST6, no combination of peak angular velocities of the index finger F2 is detected, so there are no features to use for gesture determination.

[0129] During the transition from the release state ST6 to the initial state ST0, the angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in all directions. In the latter half of the first gesture transitioning from the release state ST6 to the initial state ST0, no combination of peak angular velocities of the index finger F2 is detected, so there are no features to use for gesture detection.

[0130] Figure 16 is a diagram illustrating the relationships between multiple gestures detected in the operation input system 2 according to this embodiment. Figure 16 shows eight types of gestures G0 to G6 and G8 corresponding to the states ST0 to ST6 and ST8 described above. The direction of the arrows in the figure indicates the direction of transition from one gesture to another. Figure 17 is a side view illustrating gesture G5. Figure 18 is a side view illustrating gesture G6.

[0131] Gesture G0 is the first gesture, which starts from state ST6 where the thumb F1 and index finger F2 are separated, then moves to state ST7 where the thumb F1 is in contact with the side of the middle phalanx of the index finger F2, and then maintains this position for a certain period of time to return to the initial state ST0.

[0132] Gesture G1 is a second gesture in which, starting from the initial state ST0, the tip of the thumb F1 is moved to the left along the side of the index finger F2 while maintaining contact between the thumb F1 and the index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST1 by gesture G1.

[0133] Gesture G2 is a second gesture in which, while the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is moved to the right along the side of the index finger F2 while keeping the thumb F1 and index finger F2 in contact, and this position is maintained for a certain period of time. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST2 by gesture G2.

[0134] Gesture G3 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is extended outward beyond the back of the index finger F2 while the thumb F1 and index finger F2 are in contact. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST3 by gesture G3.

[0135] Gesture G4 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are brought into contact and contracted, moving the tip of the thumb F1 to the side of the index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST4 by gesture G4.

[0136] Gesture G5 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is pressed down while the thumb F1 and index finger F2 are in contact, causing the index finger F2 and middle finger F3 to come into contact as shown in Figure 17. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST5 by gesture G5.

[0137] Gesture G6 is a second gesture that separates the thumb F1 and index finger F2 from any given state and maintains that state for a certain period of time, as shown in Figure 18. For example, the state of the thumb F1 and index finger F2 is changed by gesture G6 from the initial state ST0, or one of states ST1 to ST5, to the released state ST6.

[0138] Gesture G8 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are separated, and then the thumb F1 taps the side of the index finger F2 a predetermined number of times (for example, twice) within a predetermined time limit.

[0139] Gestures G1-G5 and G8 are secondary gestures and are pre-associated with six types of commands related to user input. Gestures G1-G4 correspond to commands that move the cursor, etc., to the left, right, up, and down on the user input screen, respectively. Gesture G5 corresponds to a command that cancels the most recent user input on the user input screen. Gesture G8 corresponds to a command that confirms the selected item or entered information on the user input screen.

[0140] Figure 19 is a flowchart showing an example of the processing performed in the operation input system 2 according to this embodiment. Steps S101 to S108 and S110 to S113 of the processing in Figure 19 are the same as those in the flowchart of Figure 10. Therefore, the steps that differ from those in Figure 10 will be described in detail below.

[0141] In step S108, the control device 30 determines the relative finger movement of two fingers based on the combination of peak angular velocity values ​​extracted within the extraction period. The determination of finger movement is started when the first angular velocity of the thumb F1 (first signal) or the second angular velocity of the index finger F2 (second signal) is greater than or equal to a predetermined threshold (step S106: YES), thus suppressing the computational load.

[0142] After step S108, the process moves to step S201. In step S201, the control device 30 determines whether the finger movement with two fingers corresponds to the first gesture.

[0143] If the control device 30 determines that the finger movement corresponds to the first gesture, that is, that the finger movement is gesture G0 (step S201: YES), the control device 30 updates the detection flag for the first gesture stored in the memory area to ON (step S202), and the process returns to step S101.

[0144] The detection flag indicates whether or not the first gesture has been detected. The detection flag can be initialized to OFF, for example, when an operation input command is issued. Note that the method for managing the detection status of the first gesture in the operation input system 1 is not limited to using the detection flag.

[0145] If the control device 30 determines that the finger movement does not correspond to the first gesture (step S201: NO), the process proceeds to step S203.

[0146] In step S203, the control device 30 determines whether the finger movement is a second gesture. If the control device 30 determines that the finger movement is a second gesture (step S203: YES), the process proceeds to step S204.

[0147] If the control device 30 determines that the finger movement is not a second gesture (step S203: NO), the process returns to step S101.

[0148] In step S204, the control device 30 determines whether the detection flag for the first gesture is ON or not. That is, the control device 30 determines whether the first gesture was detected before the second gesture. If the control device 30 determines that the detection flag for the first gesture is ON (step S204: YES), the process proceeds to step S205.

[0149] In response, if the control device 30 determines that the detection flag for the first gesture is not ON (step S204: NO), the process returns to step S101. That is, if the control device 30 determines that the second gesture was performed before the first gesture was detected, it will not accept the operation input by the second gesture.

[0150] In step S205, the control device 30 outputs a command corresponding to the second gesture. Specifically, when the control device 30 determines that the second gesture has been performed following the first gesture, it accepts the second gesture as an operation input and outputs a command corresponding to the second gesture. Thereafter, the process proceeds to step S110. The processing from step S110 to step S113 is the same as in the case of FIG. 10.

[0151] In the case of FIG. 7 described above, the finger movement from the start time t1 of the extraction period A to the end time t4 of the extraction period B corresponds to the first gesture. Also, the finger movement in the extraction period C following the first gesture corresponds to the second gesture. In the first gesture, a pair of large amplitudes on the plus side and then the minus side occurs for the angular velocity X_1 around the X-axis of the thumb F1, and it is easy to determine the upward finger movement of the thumb F1 in the extraction period A and the downward finger movement of the thumb F1 in the extraction period B. Also, it can be easily determined that the combination of the finger movement in the extraction period A and the finger movement in the extraction period B is the first gesture.

[0152] For example, in FIG. 7, a threshold value THp is set on the plus side and a threshold value THm is set on the minus side for the angular velocity X_1 around the X-axis of the thumb F1. Here, the control device 30 (gesture detection unit 30H) classifies the period in which X_1 ≥ THp as "1", the period in which THm < X_1 < THp as "0", and the period in which X_1 ≤ THm as "-1" for the angular velocity X_1. In this case, the control device 30 can easily detect the angular velocity change pattern of "0" → "1" → "0" → "-1" → "0" in the period from time t1 to t6. Then, the control device 30 performs the detection process of the angular velocity change pattern for the other angular velocities Y_1, Z_1, X_2, Y_2, Z_2 in the same manner as the angular velocity X_1. Thereby, the control device 30 can detect the second gesture performed following the first gesture with a small computational load.

[0153] Figure 20 shows an example of an operation input screen displayed on the display device 306 according to this embodiment. In Figure 20, multiple menus are displayed on the operation input screen, and the state in which menu M1 is selected is shown. Also, a cross key K for moving the cursor position is displayed in the upper right area of ​​the operation input screen. Multiple indicators are provided around the cross key K. When a command to move the cursor position in any direction (up, down, left, or right) is issued by the operator's finger movement, the indicator corresponding to the command lights up. In Figure 20, the lower indicator of the cross key K lights up because a command to move the cursor position downwards has been issued.

[0154] Conventional input systems based on finger gestures sometimes failed to properly detect the boundary between one gesture and the next. As a result, two gestures were sometimes perceived as one, or the user's normal finger movements were mistakenly identified as a gesture, potentially leading to false detections.

[0155] For example, in a gesture involving moving a finger horizontally, if the finger almost stops midway and then moves again in the same direction, the horizontal movement gesture may be detected as one instance, or it may be determined to have been repeated twice. Also, if the user realizes a mistake midway through the movement and moves in the opposite direction, only one of the actions may be considered valid, or both may be considered valid. In this way, there was a possibility that a gesture performed by the user might be detected as a gesture different from the user's intention.

[0156] Furthermore, conventional input systems had the limitation that finger gestures could not be properly detected unless the palm and arm were in a predetermined position (for example, with the palm parallel to the ground). Addressing these problems would require the use of multiple types of sensors and extremely complex calculations.

[0157] In contrast, in the operation input system 2 according to this embodiment, the operator first performs a first gesture using the thumb F1 and index finger F2, followed by a second gesture. When the control device 30 detects the combination of the first gesture and the subsequent second gesture, it outputs a command that has been pre-associated with the second gesture.

[0158] Therefore, the operation input system 2 according to this embodiment has the advantages of high accuracy in detecting and determining gestures and low computational load. Furthermore, since the operation input system 2 is configured to detect the second gesture based on the posture at the time of the first gesture, the operator can input gestures in any stationary posture, thus increasing the degree of freedom during operation.

[0159] [Third Embodiment] The operation input system 3 according to the third embodiment will be described below. The following will mainly describe the differences from the first and second embodiments, and the common parts will be omitted or simplified in the explanation.

[0160] Figure 21 is a block diagram showing an example of the hardware configuration of the device constituting the operation input system 3 according to this embodiment. As shown in Figure 21, the smart glasses 40 include an MCU 401, a wireless communication device 402, a battery 403, and a display device 404. The operation input system 3 according to this embodiment differs from the first and second embodiments in that it further includes smart glasses 40.

[0161] Smart Glasses 40 is a wearable display that can be worn by the user as eyeglasses. Smart Glasses 40 has features such as AR (Augmented Reality) functionality, a camera function, Bluetooth and Wi-Fi connectivity, and a microphone function.

[0162] Figure 22 is a perspective view showing an example of smart glasses 40 according to this embodiment. Here, the smart glasses 40 are shown to include a spectacle frame 41 and a display device (display unit) 404 that projects the screen onto lenses 42 fitted into the spectacle frame 41. The display device 404 projects an operation input screen output from the control device 30, as shown in Figure 20.

[0163] The smart glasses 40 according to this embodiment are controlled by the control device 30 in the following manner, for example. In a mode that only displays images and videos, the control device 30 disables gesture-based operation input by the OS or application. When the control device 30 switches the execution mode to a mode that requires user input, the OS or application enables gesture-based operation input.

[0164] This allows the combination of the first gesture performed and the subsequent second gesture to be detected, enabling gesture-based input. For example, the operator can perform input according to the input screen shown in Figure 20. Subsequently, if the execution mode is switched to a mode that only displays images and videos, the OS or application will disable gesture-based input again.

[0165] According to the operation input system 3 of this embodiment, the operator attaches ring-shaped devices to their thumb F1 and index finger F2 and performs operation input by making predetermined gestures. An operation input screen is displayed on the smart glasses 40. The operation input screen is updated in real time in response to the operation input by the operator's finger gestures.

[0166] For example, if the control device 30 is a smartphone, the operator can perform the desired operation input while looking at the screen displayed on the smart glasses 40, rather than the screen displayed on the smartphone. Therefore, the operator can confirm the content of the operation input performed based on gestures from the first sensor device 10 and the second sensor device 20 using the smart glasses 40, enabling efficient operation input.

[0167] [Modified Embodiment] The present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.

[0168] For example, the first sensor device 10 and the second sensor device 20 may have an inertial measurement unit as a sensor module that includes, in addition to the gyro sensor, an acceleration sensor, a magnetic sensor, and other sensors. Alternatively, the first sensor device 10 and the second sensor device 20 may use other sensors, such as an acceleration sensor, instead of the gyro sensor as a sensor module.

[0169] Alternatively, instead of using the peak value of angular velocity during the extraction period, the integrated value of angular velocity over a predetermined period may be used. By identifying the pattern of change in the integrated value, it is possible to determine what kind of finger movement was performed. This is also true when the first sensor device 10 and the second sensor device 20 are configured to detect acceleration instead of angular velocity.

[0170] Furthermore, the setting information for the first sensor device 10 and the second sensor device 20 is set to match the finger to which they are to be attached. For this reason, it is preferable that the outer circumference of the base body 11 of the first sensor device 10 and the second sensor device 20 is marked with letters or figures indicating the finger to be attached and the direction of attachment. This allows the operator to attach the first sensor device 10 and the second sensor device 20 to each finger in the optimal position by referring to the letters or figures on the outer circumference of the first sensor device 10 and the second sensor device 20, thereby further improving the accuracy of finger movement detection.

[0171] Furthermore, in the above-described embodiment, finger movements were determined based on the relationship between the first and second angular velocities detected by the first sensor device 10 and the second sensor device 20 and a predetermined threshold. However, the method for determining finger movements is not limited to this. For example, the control device may be configured to determine finger movements by inputting the detected first and second signals into a learning model that has learned finger movements corresponding to the first angular velocity (first signal) and the second angular velocity (second signal). Using a learning model to determine finger movements has the effect of further reducing the computational load.

[0172] In the embodiments described above, it was assumed that the hand posture was largely maintained during the period from when the operator performed the first gesture until the completion of the second gesture. However, if the operator's hand posture differs significantly between the first and second gestures, the accuracy of gesture detection may decrease, or the computational load may increase. Therefore, it is preferable for the control device 30 to further include a configuration that estimates the hand posture of the operator's hands other than the first and second fingers based on the first angular velocity (first signal) and the second angular velocity (second signal). Specifically, it estimates the direction of the palm and the back of the hand. By accurately estimating the hand posture, the accuracy of detecting the first and second gestures can be improved.

[0173] Furthermore, the control device 30 may determine the second gesture on the condition that the change in hand posture after the detection of the first gesture is within a predetermined range. This can suppress a decrease in the accuracy of gesture detection. It also has the effect of reducing the computational load on the control device 30. Moreover, if the control device 30 determines that the estimated hand posture at the time of the second gesture is significantly different from the hand posture at the time of the first gesture, it is preferable to provide feedback to the operator to that effect.

[0174] In the embodiment described above, the first and second gestures were detected based on the peak value of the angular velocity during the extraction period and a common threshold. However, the threshold used to detect the first gesture may be set to a different value from the threshold used to detect the second gesture. For example, the absolute value of the threshold used to detect the second gesture may be smaller than the absolute value of the threshold used to detect the first gesture. In this case, the absolute values ​​of the angular velocity of the thumb F1 and index finger F2 in the second gesture will be smaller than the absolute values ​​of the angular velocity of each finger in the first gesture. Furthermore, since the sensitivity when detecting the second gesture will be higher than when detecting the first gesture, the operator will be able to perform the second gesture with smaller movements after detecting the first gesture than when performing the first gesture. [Explanation of Symbols]

[0175] 1, 2, 3... Operation Input System 10. First Sensor Device 20. Second Sensor Device 30.. Control device 30A...Calibration section 30B...Angular velocity calculation section 30C...Peak value extraction unit 30D... Command issuing unit 30E...Feedback Control Unit 30F...Display Control Unit 30G... Input section 30H... Gesture detection unit 40... Smart Glasses 101...MCU 102...Sensor Module 103... Wireless communication device 104...Feedback Module 105A... Tightening mechanism 105B...Vibration mechanism 105... Battery 201...MCU 202...Sensor Module 203... Wireless communication device 204...Feedback Module 205... Battery 301 Processor 302...RAM 303···ROM 304...Storage 305...Communication Interface 306...Display device 307...Input device 401...MCU 402... Wireless communication equipment 403...Display device 404... Battery

Claims

1. A first sensor device having a ring-shaped base that can be attached to any first finger among the five fingers of one hand of an operator, and which outputs a first signal corresponding to the movement of the first finger, A second sensor device having an annular base that can be attached to a second finger, which is different from the first finger, among the five fingers, and which outputs a second signal corresponding to the movement of the second finger, A control device that determines the relative finger movements of the first finger and the second finger based on the first signal and the second signal, Equipped with, Each of the first sensor device and the second sensor device includes a plurality of sensor modules and at least one feedback module that performs a notification operation to the operator. The feedback module is an operation input system having one of the following: a tightening mechanism for tightening the first finger or the second finger, and a vibration mechanism for applying vibration to the first finger or the second finger.

2. The first sensor device includes a first wireless communication device that transmits the first signal to the control device by wireless communication, The second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication. The operation input system according to claim 1.

3. Multiple sensor modules are arranged at approximately equal intervals around the base body. The operation input system according to claim 1.

4. The outer circumference of the base body is marked with letters or figures indicating the finger to be worn and the direction of wear. The operation input system according to claim 1.

5. The control device determines the mounting status of the first sensor device and the second sensor device based on the first signal and the second signal. The operation input system according to claim 1.

6. The control device performs calibration of the first signal and the second signal when the first sensor device and the second sensor device are started. The operation input system according to claim 1.

7. The aforementioned tightening mechanism has an inflatable or deflated airbag. The operation input system according to claim 1.

8. The aforementioned tightening mechanism has an expandable or contractible dielectric member. The operation input system according to claim 1.

9. The vibration mechanism has an expandable or contractible piezoelectric element. The operation input system according to claim 1.

10. The vibration mechanism has a vibrating electromagnetic drive unit, The operation input system according to claim 1.

11. The operation input system according to claim 1, wherein each of the first sensor device and the second sensor device includes a plurality of feedback modules corresponding to the plurality of sensor modules, respectively.

12. A first sensor device is attached to any first finger among the five fingers of one hand of the operator, and detects a first signal corresponding to the movement of the first finger. A second sensor device is attached to the second finger, which is different from the first finger, among the five fingers mentioned above, and detects a second signal corresponding to the movement of the second finger. A control device that controls the first sensor device and the second sensor device, respectively, Equipped with, Each of the first sensor device and the second sensor device includes a plurality of sensor modules and at least one feedback module that performs a notification operation to the operator. The feedback module includes one of the following: a tightening mechanism for tightening the first finger or the second finger, and a vibration mechanism for applying vibration to the first finger or the second finger. The control device is Based on the first signal and the second signal, the relative finger movements of the first finger and the second finger are detected. On the condition that the aforementioned finger movement is determined to be a predetermined first gesture, it is determined whether the finger movement performed following the first gesture is a predetermined second gesture. If it is determined that the finger movement is the second gesture, the command corresponding to the second gesture is output. Operation input system.

13. The control device starts determining the finger movement when the first signal or the second signal exceeds a threshold. The operation input system according to claim 12.

14. The threshold value in the determination of the first gesture is greater than the threshold value in the determination of the second gesture. The operation input system according to claim 13.

15. The control device estimates the position of the operator's hand other than the first and second fingers based on the first and second signals. The operation input system according to claim 12.

16. The control device determines the second gesture on the condition that the change in the hand's posture after the detection of the first gesture is within a predetermined range. The operation input system according to claim 15.

17. The control device determines the finger movement by inputting the detected first signal and second signal into a learning model that has learned the finger movement corresponding to the first signal and second signal. The operation input system according to claim 12.

18. The first finger is the thumb, and the second finger is the index finger. The first gesture is the action of separating the thumb and the index finger, and then bringing the thumb into contact with the side of the index finger. The operation input system according to claim 12.

19. The second gesture is an action in which the thumb is moved to the left, right, forward, and backward directions along the side of the index finger while the thumb and index finger are in contact. The operation input system according to claim 18.

20. The second gesture is an action in which the thumb is pressed down while the thumb and index finger are in contact, causing the thumb to come into contact with the index finger and middle finger. The operation input system according to claim 18.

21. The second gesture is the action of separating the thumb and index finger that are in contact. The operation input system according to claim 18.

22. The second gesture is the action of bringing the thumb into contact with the index finger after performing the upward and downward swings of the thumb a predetermined number of times in a row. The operation input system according to claim 18.

23. The control device outputs the command to move the position of the pointer or cursor displayed on the operation input screen to the left, right, up, or down, in response to the second gesture. The operation input system according to claim 18.

24. The control device, in response to the second gesture, outputs the command on the operation input screen to cancel the operation input that was performed immediately before. The operation input system according to claim 18.

25. The control device outputs the command to reset the flag indicating the detection status of the first gesture in response to the second gesture. The operation input system according to claim 18.

26. The control device outputs the command that determines the operation input performed immediately before the second gesture on the operation input screen, in response to the second gesture. The operation input system according to claim 18.

27. The first sensor device includes a first wireless communication device that transmits the first signal to the control device by wireless communication, The second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication. The operation input system according to claim 18.

28. A wearable display including an eyeglass frame and a display unit that projects an operation input screen output from the control device onto a lens fitted into the eyeglass frame. The operation input system according to claim 18, further comprising:

Images