Gesture recognition device and gesture recognition method

Abstract

The purpose of the present invention is to provide a gesture recognition device that can recognize a hand gesture even when the brightness of a room is insufficient and when a hand is hidden by an obstacle. The present disclosure is a gesture recognition device that uses electromagnetic waves that have a frequency in or above the terahertz range and comprises: a band that is wound around a portion of the body of a user that includes hand nerves and hand muscles; a sensor that is fixed by the band and closely contacts the portion of the body; and a learning circuit. The sensor comprises: a transmitter that transmits electromagnetic waves into the portion of the body; and a receiver that receives electromagnetic waves that are reflected inside the portion of the body and have returned to the sensor. The learning circuit uses a learning model to infer a hand gesture made by the user during propagation of the electromagnetic waves inside the portion of the body from a reception waveform for the electromagnetic waves received by the sensor. The learning model infers the hand gesture made by the user on the basis of the reception waveform for the electromagnetic waves from the movement of the hand nerves and the hand muscles of the user through which the electromagnetic waves have propagated.

Description

Gesture recognition device and gesture recognition method

[0001] The present disclosure relates to a gesture recognition device and a gesture recognition method that use electromagnetic waves with frequencies in the terahertz range or higher.

[0002] Research and development is being promoted on the metaverse, a three-dimensional virtual space, digital twins, and cyber-physical fusion. New services based on these technologies are also being put into practical use. Hand gesture recognition technology is attracting attention as an input interface (IF), a means of transferring real-world information to the virtual world.

[0003] Non-Patent Document 1 discloses a hand and finger detection method used in a hand gesture recognition system, in which hand gestures are recognized based on images of hands captured by a monocular camera.

[0004] Yuya Nemoto and Katsuhito Nakajima, "High-speed finger detection for hand gesture recognition using a monocular camera," Proceedings of the 76th National Conference of the Information Processing Society of Japan, Vol. 2, pp. 77-78 (2014).

[0005] However, the above-mentioned method using a camera has the problem that, although the recognition accuracy is high when the image is clear, the recognition accuracy deteriorates when the room is not bright enough or when the hand is hidden by an obstacle.

[0006] In order to solve the above-mentioned problems, a first object of the present disclosure is to provide a gesture recognition device that can recognize hand gestures even when the room is not bright enough or when the hand is hidden by an obstacle.

[0007] A second object of the present disclosure is to provide a gesture recognition method that can recognize hand gestures even when the room is poorly lit and when the hand is hidden behind an obstacle.

[0008] A first aspect of the present disclosure is a gesture recognition device that uses electromagnetic waves having a frequency in the terahertz range or higher, comprising: a band that is wrapped around a part of a user's body that includes the nerves and muscles of the hand; a sensor that is fixed by the band and is in close contact with the body part; and a learning circuit, wherein the sensor comprises: a transmitter that transmits the electromagnetic waves to the body part; and a receiver that receives the electromagnetic waves that have been reflected within the body part and returned to the sensor, and the learning circuit is configured to execute a process of estimating, using a learning model, a hand gesture performed by the user while the electromagnetic waves were propagating through the body part from the received waveform of the electromagnetic waves received by the sensor, and the learning model is preferably a learning model that estimates a hand gesture performed by the user from the movement of the nerves and muscles of the user's hand through which the electromagnetic waves have propagated, based on the received waveform of the electromagnetic waves.

[0009] A second aspect is a gesture recognition method using electromagnetic waves having a frequency in the terahertz range or higher, comprising: a sensor transmitting the electromagnetic waves to a part of a user's body including the nerves and muscles of the hand; the sensor receiving the electromagnetic waves that have been reflected within the body part and returned to the sensor; and estimating, from the received waveform of the received electromagnetic waves, a hand gesture performed by the user while the electromagnetic waves were propagating within the body part using a learning model, wherein the learning model is preferably a learning model that estimates the hand gesture performed by the user from the movement of the nerves and muscles of the user's hand through which the electromagnetic waves have propagated, based on the received waveform of the electromagnetic waves.

[0010] In the gesture recognition device disclosed herein, a sensor is attached to a part of a user's body that includes the nerves and muscles of the hand. The gesture recognition device also transmits electromagnetic waves with frequencies in the terahertz range or higher toward the user's body part from the sensor, and estimates hand gestures made by the user while the electromagnetic waves propagate through the user's body part. This makes it possible to provide a gesture recognition device and a gesture recognition method that can recognize hand gestures even when the room is poorly lit or when the hand is hidden by an obstacle.

[0011] FIG. 1 is a diagram illustrating a gesture recognition device in a state where a sensor is worn by a user according to a first embodiment of the present disclosure. FIG. 2 is a view taken along the arrows A-A in FIG. 1. FIG. 3 is a block diagram illustrating an example functional configuration of a sensor according to the first embodiment of the present disclosure. FIG. 4 is a flowchart of a process executed by a sensor when calibrating a learning model according to the first embodiment of the present disclosure. FIG. 5 is a flowchart of a process executed by a sensor when estimating a hand gesture according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating a gesture recognition device according to a modified example of the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating a gesture recognition device in a state where a sensor is worn by a user according to a second embodiment of the present disclosure. FIG. 8 is a diagram illustrating a gesture recognition device according to a modified example of the second embodiment of the present disclosure.

[0012] Embodiments of the present disclosure will be described with reference to the drawings. The same or corresponding components will be designated by the same reference numerals, and repeated description may be omitted.

[0013] 1 is a diagram illustrating a gesture recognition device 100 according to a first embodiment of the present disclosure in a state in which a sensor 2 is worn by a user. Also, FIG. 2 is a view taken along the line AA in FIG.

[0014] The gesture recognition device 100 includes a band 1 that is wrapped around a part 20 of a user's body, such as an arm or wrist, that includes the nerves and muscles of the hand, a display 4, and a sensor 2 that is fixed by the band 1 and in close contact with the user's skin. In particular, the gesture recognition device 100 in which the sensor 2 is worn on the wrist as shown in Fig. 1 is preferably provided as a wristwatch-type device, such as a smartwatch.

[0015] The sensor 2 transmits terahertz waves 10 toward the user's skin, which is in close contact with the sensor 2. The transmitted terahertz waves penetrate the user's skin and propagate through the user's body part 20, such as nerves and muscles. A portion of the transmitted terahertz waves is reflected by the user's body part 20, such as nerves and muscles, and returns to the sensor 2. The sensor 2 receives the returned terahertz waves 10. The sensor 2 then executes a process (hereinafter referred to as an estimation process) to estimate, from the received waveform of the terahertz waves 10, a hand gesture made by the user while the terahertz waves 10 were propagating through the body part 20. In the estimation process, a calibrated learning model 60 calibrated by a method described below is used.

[0016] The display 4 is a display device that displays the results of the estimation process performed by the sensor 2 to the user.

[0017] The sensor 2 does not necessarily have to be attached to the display 4, but may be fixed directly or indirectly by the band 1 so as to be in close contact with the user's skin.

[0018] In this manner, in the present disclosure, a user's hand gestures are recognized using terahertz waves 10. By using terahertz waves 10, movements of the user's nerves, muscles, and the like can be observed more precisely than when electromagnetic waves with frequencies lower than terahertz waves 10 are used, and it becomes possible to estimate the hand gestures performed by the user.

[0019] FIG. 3 is a block diagram illustrating an example of a functional configuration of the sensor 2 according to the first embodiment of the present disclosure.

[0020] The transmitter 21 of the sensor 2 includes a laser light source and a transmitting antenna, and transmits terahertz waves 10 toward the user's skin. The frequency of the terahertz waves 10 may be a single frequency, a continuous frequency, or multiple discrete frequencies. By using terahertz waves 10 of continuous or discrete frequencies, the receiver 22 can obtain received waveforms for multiple frequencies, improving the reliability of the data and increasing the accuracy of hand gesture estimation.

[0021] The receiver 22 includes a receiving antenna, an analog processing circuit, and a digital processing circuit. The receiver 22 receives the terahertz waves 10 that have returned to the sensor 2, and this includes terahertz waves 10 that have been transmitted from the transmitter 21 and then reached the receiver 22 directly without propagating through the user's body part 20. The receiver 22 removes the received waveform of this direct terahertz wave 10, and then sends only the received waveform of the terahertz wave 10 that has propagated through the user's body part 20 to the learning circuit 23.

[0022] Although the number of transmitting antennas and receiving antennas is not limited, using an array antenna consisting of multiple antennas enables beamforming of the terahertz waves 10 during transmission and reception. This makes it possible to form a highly directional beam, improving the reception efficiency of the terahertz waves 10.

[0023] The learning circuit 23 pre-stores a learning model 50 that estimates the hand gesture performed by the user from the movement of the nerves and muscles in the user's hand through which the terahertz waves 10 have propagated, based on the received waveform of the input terahertz waves 10.

[0024] Methods for creating the learning model 50 include known methods such as random forest, SVM (Support Vector Machine), K-nearest neighbor method, and neural network.

[0025] Here, the learning model 50 is not learned using the user who actually uses the gesture recognition device 100 as a teacher, and is therefore calibrated when the user starts using the gesture recognition device 100.

[0026] The learning circuit 23 calibrates the pre-stored learning model 50 based on the hand gestures performed by the user of the gesture recognition device 100. Here, the learning model 50 is re-trained using the user's hand gestures and the received waveform of the terahertz waves 10 measured when the hand gestures are performed as training data. This makes it possible to generate a calibrated learning model 60 targeted at the user who actually uses the gesture recognition device 100.

[0027] The learning circuit 23 performs the above-described estimation process on the received waveform of the terahertz wave 10 received from the receiver 22 using the calibrated learning model 60 .

[0028] The processing performed by the learning circuit 23 of the gesture recognition device 100 may be executed by a program using a computer including a CPU and memory and storing a program in the memory. Alternatively, the processing may be executed by a program using an integrated circuit such as an FPGA (Field Programmable Gate Array). The program may be provided by being recorded on a storage medium or via a network. This point is common to all of the following embodiments.

[0029] FIG. 4 is a flowchart of a process executed by the sensor 2 when calibrating the learning model 50 according to the first embodiment of the present disclosure. First, the process is started. Next, a user who starts using the gesture recognition device 100 is instructed to perform a predetermined hand gesture (step S01). Next, while the user is performing the hand gesture, the transmitter 21 transmits terahertz waves 10 toward the user's skin, which is in close contact with the sensor 2 (step S02). Next, the receiver 22 receives the terahertz waves 10 and acquires the terahertz waves 10 for training data (step S03). Next, the learning circuit 23 calibrates the pre-stored learning model 50 based on the received waveform of the terahertz waves 10 for training data (step S04). Next, it is determined whether the calibration by the learning circuit 23 is complete and the calibrated training model 60 has been generated (step S05). If the calibration is deemed complete, the process ends. On the other hand, if the calibration is not deemed complete, the process returns to step S01 and attempts to acquire the terahertz waves 10 for training data again.

[0030] If there are a plurality of predetermined hand gestures, the process of FIG. 4 may be repeated while changing the hand gesture instructed to the user in step S01.

[0031] 5 is a flowchart of a process executed by sensor 2 when estimating a hand gesture according to the first embodiment of the present disclosure. First, the process is started. Next, transmitter 21 transmits terahertz waves 10 toward the user's skin that is in close contact with sensor 2 (step S11). Next, receiver 22 receives terahertz waves 10 (step S12). Furthermore, learning circuit 23 performs estimation processing using calibrated learning model 60, and estimates the user's hand gesture from the received waveform of terahertz waves 10 (step S13). Finally, the process ends.

[0032] As described above, in the gesture recognition device 100 of the present disclosure, the sensor 2 is attached to the user's body part 20, which includes the nerves and muscles of the hand. The gesture recognition device 100 transmits terahertz waves 10 from the sensor 2 toward the user's body part 20, and estimates a hand gesture performed by the user while the terahertz waves 10 propagate through the user's body part 20. The gesture recognition device 100 of the present disclosure does not use a camera as in conventional technologies, and therefore estimation accuracy is not affected by room brightness. Furthermore, in the gesture recognition device 100 of the present disclosure, the sensor 2 is attached to the user's arm or wrist, and hand gestures are estimated from the movements of the user's nerves and muscles in the hand, so that the estimation is not affected by the hand itself being hidden by an obstacle.

[0033] Therefore, according to the present disclosure, it is possible to provide a gesture recognition device and a gesture recognition method that can recognize hand gestures even when the room is not bright enough and when the hand is hidden by an obstacle.

[0034] In hand gesture recognition technology, a method is also known in which multiple motion detection devices are attached to the back of the hand or fingers to recognize hand gestures performed by a user. However, this method has the problem that the devices are attached directly to the back of the hand or fingers, which interferes with hand movements. In contrast, the gesture recognition device 100 of the present disclosure is advantageous in that it does not require the sensor 2 to be attached to the back of the hand or fingers and does not interfere with hand movements.

[0035] <First Variation of First Embodiment> The learning circuit 23 does not necessarily have to be built into the sensor 2, but may be arranged outside the sensor 2. By arranging it outside the sensor 2, it becomes unnecessary for the sensor 2 to calibrate the learning model 50 and to perform processing related to the estimation of hand gestures, thereby reducing the load. This is common to all of the following embodiments.

[0036] <Second Modification of First Embodiment> It has been described that the calibrated learning model 60 is used in the above-described estimation process performed by the learning circuit 23. However, in cases where the estimation accuracy of the pre-stored learning model 50 is high, the estimation process may be performed using the learning model 50. This eliminates the need to calibrate the learning model 50, thereby reducing the load on the learning circuit 23. This is common to all of the following embodiments.

[0037] <Third Modification of First Embodiment> Note that, as shown in FIG. 6 , the number of sensors 2 may be multiple. FIG. 6 is a diagram illustrating a gesture recognition device 100 according to a third modification of the first embodiment of the present disclosure. Here, the view is shown from the same direction as FIG. 2 . The multiple sensors 2 are attached so as to be in close contact with a body part 20 of the user. Note that the attachment positions of the sensors 2 are not limited, but it is preferable that the sensors 2 be attached isotropically to the body part 20 of the user, as shown in FIG. 6 . By arranging multiple sensors 2, it is possible to detect the movement of the nerves and muscles of the user's hand from various directions, thereby further improving the accuracy of hand gesture estimation. This is common to all of the following embodiments.

[0038] <Fourth Modification of First Embodiment> The terahertz waves 10 received by receiver 22 are terahertz waves 10 that have propagated through multiple tissues, such as the user's skin, nerves, and muscles, and finally returned to receiver 22. To improve the accuracy of hand gesture estimation, information on the received waveform of terahertz waves 10 at specific tissues may be extracted from the received terahertz waves 10. For example, by mapping the intensity of terahertz waves 10 at the user's skin on a two-dimensional plane, the detailed condition of the skin during hand gestures can be analyzed. This is expected to improve the accuracy of hand gesture estimation. This is common to all of the following embodiments.

[0039] <Fifth Modification of First Embodiment> In the above description, the body part 20 to which the sensor 2 is attached is a body part 20 including the nerves and muscles of the user's hand, such as the arm or wrist. However, the body part 20 may also be a body part including the nerves and muscles of the user's foot, such as the leg or ankle. The gesture recognition device 100 can transmit terahertz waves 10 toward the user's leg, ankle, or the like, and estimate a foot gesture performed by the user while the terahertz waves 10 propagate through the user's leg, ankle, or the like. This is common to all of the following embodiments.

[0040] Sixth Modification of First Embodiment It has been explained that the reason for using terahertz waves 10 in the present disclosure is that it allows for more precise observation of the movements of a user's nerves, muscles, and the like, and for estimating hand gestures, compared to when electromagnetic waves with frequencies lower than terahertz waves 10 are used. It should be noted that any electromagnetic wave with a frequency equal to or higher than terahertz waves 10, not limited to terahertz waves 10, can observe the movements of a user's nerves, muscles, and the like, and can estimate hand gestures as precisely as or better than terahertz waves 10. Therefore, in the present disclosure, it is sufficient to use electromagnetic waves with a frequency equal to or higher than terahertz waves 10, and in that case, the same effects as those described above can be obtained. This is common to all of the following embodiments.

[0041] Second Embodiment In this embodiment, the gesture recognition device 100 further includes a reflector 3. In the following, changes from the first embodiment will be described.

[0042] 7 is a diagram illustrating the gesture recognition device 100 according to the second embodiment of the present disclosure when worn by a user, as viewed from the same direction as in FIG.

[0043] The reflector 3 is made of a material that is reflective to the terahertz waves 10. The reflector 3 is disposed inside the band 1. The reflector 3 is disposed in a position where it can return the terahertz waves 10 transmitted from the sensor 2 to the sensor 2 when the sensor 2 is worn by a user. In FIG. 7 , the reflector 3 is disposed in a direction directly facing the sensor 2.

[0044] As in the first embodiment, the sensor 2 transmits the terahertz wave 10 toward the user's skin that is in close contact with the sensor 2 .

[0045] Here, the transmitted terahertz waves 10 pass through the user's skin and propagate within the user's body part 20, such as nerves and muscles, but as explained in embodiment 1, a portion of the waves is reflected within the user's body part 20 and returns to the sensor 2.

[0046] In this embodiment, the reflector 3 is disposed inside the band 1, and therefore the terahertz waves 10 that have propagated through the user's body part 20 are also reflected by the reflector 3 and return to the sensor 2. By disposing the reflector 3 in this manner, the terahertz waves 10 that are diffused in various directions can be returned to the sensor 2, and the reception efficiency of the terahertz waves 10 in the receiver 22 can be improved. This allows for improved accuracy in estimating hand gestures compared to the first embodiment.

[0047] <First Modification of Second Embodiment> Note that, as shown in Fig. 8 , there may be a plurality of reflectors 3. Fig. 8 is a diagram showing a gesture recognition device 100 according to a modification of the second embodiment of the present disclosure. By isotropically attaching the reflectors 3 to the user's body part 20 in this manner, the effect of returning the diffusing terahertz waves 10 to the sensor 2 can be further enhanced.

[0048] Second Modification of Second Embodiment By using a material inside band 1 that reflects terahertz waves 10, terahertz waves 10 can be reflected not only by reflector 3 but also by band 1, thereby further improving the reception efficiency of terahertz waves 10. This further improves the accuracy of hand gesture estimation.

[0049] The present disclosure is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the present disclosure. Furthermore, the embodiments and their modifications may be implemented in appropriate combinations, in which case the combined effects can be obtained.

[0050] 1 Band, 2 Sensor, 3 Reflector, 4 Display, 10 Terahertz wave, 20 Body part, 21 Transmitter, 22 Receiver, 23 Learning circuit, 50 Learning model, 60 Calibrated learning model, 100 Gesture recognition device

Claims

1. A gesture recognition device using electromagnetic waves having a frequency in the terahertz range or higher, comprising: a band that is wrapped around a part of a user's body that includes the nerves and muscles of the hand; a sensor that is fixed by the band and in close contact with the body part; and a learning circuit, wherein the sensor comprises: a transmitter that transmits the electromagnetic waves to the body part; and a receiver that receives the electromagnetic waves that have been reflected within the body part and returned to the sensor, wherein the learning circuit is configured to execute a process of estimating, from a received waveform of the electromagnetic waves received by the sensor, a hand gesture performed by the user while the electromagnetic waves were propagating through the body part, using a learning model, and the learning model is a learning model that estimates a hand gesture performed by the user from the movement of the nerves and muscles of the user's hand through which the electromagnetic waves have propagated, based on the received waveform of the electromagnetic waves.

2. The gesture recognition device of claim 1, wherein the learning model is a calibrated calibrated learning model, and the learning circuit is further configured to execute the following processes: instructing a user to perform a predetermined hand gesture; causing the transmitter to transmit the electromagnetic waves to the body part while the user is performing the predetermined hand gesture; causing the receiver to receive the electromagnetic waves reflected within the body part and returning to the sensor, and use them as electromagnetic waves for teacher data; and calibrating a pre-stored learning model based on the received waveform of the electromagnetic waves for teacher data to use as the calibrated learning model.

3. The gesture recognition device according to claim 1 or 2, further comprising a reflector having reflectivity to the electromagnetic waves, the reflector being arranged inside the band and configured to perform a process of reflecting the electromagnetic waves transmitted from the sensor and returning them to the sensor.

4. A gesture recognition method using electromagnetic waves having a frequency in the terahertz range or higher, comprising: a sensor transmitting the electromagnetic waves to a part of a user's body including the nerves and muscles of the hand; the sensor receiving the electromagnetic waves that have been reflected within the body part and returned to the user; and estimating, from the received waveform of the electromagnetic waves, a hand gesture performed by the user while the electromagnetic waves were propagating through the body part using a learning model, wherein the learning model is a learning model that estimates, based on the received waveform of the electromagnetic waves, a hand gesture performed by the user from the movement of the nerves and muscles of the user's hand through which the electromagnetic waves have propagated.

Images