Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program, that enable realizing a higher degree of freedom in operability. An information processing device according to the present invention estimates a finger and hand shape of a user and a force state of a hand of the user on the basis of myoelectric potential data corresponding to movement of the hand and the fingers of the user in a state of gripping or supporting a given real object, and executes processing corresponding to operations performed by the user. Further, the information processing device detects that the user has performed an initial operation, on the basis of the finger and hand shape of the user and the force state of the fingers of the user. The present technology can be applied to, for example, a terminal operation system using a given real object.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable operability with a higher degree of freedom to be realized.

[0002] 2. Description of the Related Art Conventionally, there has been progress in the development of a technology for measuring the myoelectric potential of a user's arm and operating an information processing terminal in accordance with the movements of the user's hand and fingers.

[0003] For example, Patent Document 1 discloses a motion recognition method using a grasped object, which estimates the state of a user's wrist through a writing action using the grasped object, estimates the joint movements of the user's body parts associated with the wrist through the writing action, and estimates the state of the grasped object from the wrist state and joint movements.

[0004] JP 2015-001978 A

[0005] Incidentally, as disclosed in the above-mentioned Patent Document 1, there is a demand not only for the recognition of content written using a pen-shaped, elongated object, but also for the realization of a high degree of freedom in operability, for example, by enabling the use of a variety of arbitrary objects.

[0006] The present disclosure has been made in view of such circumstances, and aims to make it possible to realize operability with a higher degree of freedom.

[0007] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires measurement data corresponding to the movement of a user's hand and fingers while grasping or supporting any real object, and a calculation unit that estimates the shape of the user's fingers and the force state of the hand based on the measurement data and executes processing corresponding to the user's operation.

[0008] An information processing method or program according to one aspect of the present disclosure includes acquiring measurement data corresponding to the movement of a user's hand and fingers while gripping or supporting any real object, estimating the shape of the user's fingers and the force state of the hand based on the measurement data, and executing processing corresponding to the user's operation.

[0009] In one aspect of the present disclosure, measurement data corresponding to the movement of a user's hand and fingers while grasping or supporting any real object is acquired, and based on the measurement data, the shape of the user's fingers and the force state of the hand are inferred, and processing corresponding to the user's operation is performed.

[0010] 1 is a block diagram showing an example configuration of an embodiment of a terminal operation system to which the present technology is applied; FIG. 2 is a block diagram showing an example configuration of an electromyography measuring device and an information processing terminal; FIG. 3 is a diagram explaining a recognition model used to identify a user's hand shape; FIG. 4 is a diagram explaining identification of a force state based on a threshold; FIG. 5 is a diagram explaining identification of a force state using a recognition model; FIG. 6 is a diagram explaining detection of an initial operation; FIG. 7 is a diagram explaining calibration for setting a threshold; FIG. 8 is a diagram explaining hand shape, operation mode, and operation content; FIG. 9 is a diagram explaining hand shape, operation mode, and operation content; FIG. 10 is a flowchart explaining electromyography measurement processing; FIG. 11 is a flowchart explaining operation execution processing; FIG. 12 is a diagram explaining an example of a usage example using a smartwatch; FIG. 13 is a diagram explaining an example of a usage example when a user cooks; FIG. 14 is a diagram explaining an example of changing the display size of an operation guide over time; FIG. 15 is a diagram explaining an example of an operation on a keyboard; FIG. 16 is a block diagram showing an example configuration of an embodiment of a computer to which the present technology is applied.

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Configuration Example of Terminal Operation System> FIG. 1 is a diagram showing a configuration example of an embodiment of a terminal operation system to which the present technology is applied.

[0013] As shown in FIG. 1, the terminal operation system 11 includes a myoelectric potential measuring device 12 and an information processing terminal 13 .

[0014] The myoelectric potential measuring device 12 measures the myoelectric potential of the arm of a user who uses the terminal operation system 11, and transmits myoelectric potential data obtained as a result of measuring the user's myoelectric potential through communication with the information processing terminal 13. Although Fig. 1 shows a ring-shaped myoelectric potential measuring device 12 worn on the user's right arm, the myoelectric potential measuring device 12 may be worn anywhere on the user's arm from the shoulder to the wrist on either the left or right side.

[0015] For example, the myoelectric potential measuring device 12 may be configured with multiple myoelectric potential sensors that are in close contact with the skin surface, and each of the myoelectric potential sensors may have at least three electrodes: a positive electrode, a negative electrode, and a reference electrode. The electrodes of the myoelectric potential sensors may be configured so that the reference electrode is positioned on a straight line between the positive and negative electrodes, or may be configured so that the reference electrode is positioned elsewhere. The myoelectric potential measuring device 12 may also be configured so that multiple myoelectric potential sensors are arranged in a grid pattern.

[0016] The information processing terminal 13 is an electronic device that is operated by a user of the terminal operation system 11. In the example shown in Fig. 1, a head-mounted display for AR (Augmented Reality) or VR (Virtual Reality) is illustrated as an example of the information processing terminal 13, but other devices such as a smartphone or a smartwatch may also be used. The information processing terminal 13 communicates with the electromyogram measuring device 12, receives electromyogram data transmitted from the electromyogram measuring device 12, and executes processing according to the user's operation on the information processing terminal 13 based on the electromyogram data.

[0017] For example, when a user grasps or supports any real object 21 and applies force to multiple fingers to perform an initial movement, the myoelectric potential of the user's arm corresponding to the force applied to each finger is measured by the myoelectric potential measuring device 12, and the information processing terminal 13 detects that the initial movement has been performed based on the myoelectric potential data. Then, based on the myoelectric potential data, the information processing terminal 13 recognizes the shape of the user's fingers (shape of the hand and fingers) and the force state of the user's fingers (which of the five fingers is tensing, the strength of the tension, etc.), and can determine the operation content of the operation performed by the user on the information processing terminal 13 based on the inferred result.

[0018] This allows a user wearing the myoelectric potential measuring device 12 to grasp or support any real object 21 with a predetermined hand and finger shape and apply pressure with the desired fingers, thereby performing various operations on the information processing terminal 13.

[0019] 1, the target of user operation is the information processing terminal 13 that recognizes the user's hand shape and finger force state and performs processing to determine the operation content of the operation performed by the user, but the target of user operation may be an electronic device other than the information processing terminal 13 that performs such processing. In other words, the information processing terminal 13 can notify the other electronic device of the operation content so that processing according to the user operation is executed.

[0020] FIG. 2 is a block diagram showing an example of the configuration of the myoelectric potential measuring device 12 and the information processing terminal 13.

[0021] As shown in FIG. 2, the myoelectric potential measuring device 12 is configured to include eight myoelectric potential sensors 31-1 to 31-8, a signal acquisition unit 32, a signal processing unit 33, and a communication unit 34, and the information processing terminal 13 is configured to include a communication unit 41, a memory unit 42, and a calculation unit 43.

[0022] 1, the myoelectricity sensors 31-1 to 31-8 are arranged at predetermined intervals around the user's arm when the myoelectricity measuring device 12 is worn on the user's arm, and output myoelectricity signals (electrical signals obtained by measuring and amplifying the potential difference between the positive electrode, negative electrode, and reference electrode) corresponding to the weak electric field generated in the muscles of the user's arm at each position. Note that, when there is no need to distinguish between the myoelectricity sensors 31-1 to 31-8, they will hereinafter be simply referred to as the myoelectricity sensors 31. Furthermore, in the example shown in FIG. 2, eight myoelectricity sensors 31-1 to 31-8 are used, but the number of myoelectricity sensors 31 provided in the myoelectricity measuring device 12 may be eight or more or eight or less.

[0023] The signal acquiring unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31 - 1 to 31 - 8 and supplies them to the signal processing unit 33 .

[0024] The signal processing unit 33 performs signal processing such as noise removal and Fourier transform on the myoelectric potential signal supplied from the signal acquiring unit 32, for example, by a circuit using an operational amplifier, a high-pass filter, or by digital processing. The signal processing unit 33 then supplies the myoelectric potential data (a waveform indicating changes in myoelectric potential according to the movements of the user's hand and fingers) obtained as a result of the signal processing on the myoelectric potential signal to the communication unit 34.

[0025] The communication unit 34 communicates with the communication unit 41 of the information processing terminal 13 and transmits the myoelectric potential data supplied from the signal processing unit 33 .

[0026] The communication unit 41 communicates with the communication unit 34 of the myoelectric potential measuring device 12 to receive myoelectric potential data and supplies the data to the calculation unit 43. The communication unit 34 and the communication unit 41 can communicate using a wired cable, a wireless LAN (Local Area Network), Bluetooth (registered trademark), or the like.

[0027] The storage unit 42 is a memory or storage that stores pre-registered data such as the motion form of an initial operation, the correspondence between the user's finger shapes and operation modes, and operation details for each operation mode.

[0028] The calculation unit 43 is configured with a CPU (Central Processing Unit), an IC (Integrated Circuit), etc., and performs calculations using a machine learning model, a recognition model, an algorithm, etc. to estimate the hand shape and hand force state of the user based on the myoelectric potential data supplied from the communication unit 41. Then, the calculation unit 43 acquires an operation mode corresponding to the hand shape of the user from the registered data in the storage unit 42, and executes processing in accordance with the operation content corresponding to the force state of the user's hand in that operation mode.

[0029] As shown in the figure, the calculation unit 43 is configured to include a hand shape identification unit 51 , a force state identification unit 52 , an initial action detection unit 53 , an operation mode selection unit 54 , and an operation execution unit 55 .

[0030] The hand posture identification unit 51 identifies the user's hand posture based on the myoelectric potential data supplied from the communication unit 41, and supplies the recognition result indicating the user's hand posture to the initial action detection unit 53 and the operation mode selection unit 54.

[0031] For example, as shown in Fig. 3, myoelectric potential data for various hand and finger postures, such as an open hand posture, a hand posture with the entire hand gripping an object, and a hand posture with certain fingers pinching an object, can be obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8. The hand and finger posture identification unit 51 can identify the user's hand and finger posture by using a recognition model generated by machine learning (e.g., k-nearest neighbor method, support vector machine, neural network, etc.) using the myoelectric potential data as input. That is, the hand and finger posture identification unit 51 can input the myoelectric potential data supplied from the communication unit 41 to the recognition model and output a recognition result indicating the user's hand and finger posture recognized from the myoelectric potential data.

[0032] The force state identification unit 52 identifies the force state of the user's finger based on the electromyography data supplied from the communication unit 41, and supplies the recognition result indicating the force state of the user's finger to the initial action detection unit 53 and the operation execution unit 55.

[0033] 4, the force state determination unit 52 can determine the force state of the user's finger in accordance with a threshold value for the myoelectric potential data. That is, the force state of the user's finger is determined depending on whether the peak value of the myoelectric potential data obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 is equal to or greater than a preset threshold value indicated by the dashed dotted line.

[0034] 4, when all of the peak values ​​of the myoelectric potential data corresponding to the myoelectric potential sensors 31-1 to 31-8 are equal to or less than the threshold, the force state is determined to be one in which the hand is at rest with the hand spread out on the desk. When the peak values ​​of the myoelectric potential data corresponding to the myoelectric potential sensors 31-1, 31-3, and 31-6 to 31-8 among the myoelectric potential sensors 31-1 to 31-8 are equal to or greater than the threshold, the force state is determined to be one in which the hand is spread out on the desk with the index finger tensed. When the peak values ​​of the myoelectric potential data corresponding to the myoelectric potential sensors 31-1 to 31-7 among the myoelectric potential sensors 31-1 to 31-8 are equal to or greater than the threshold, the force state is determined to be one in which the hand is spread out on the desk with the thumb tensed.

[0035] 5 , the force state identification unit 52 can identify the force state of the user's fingers by using a recognition model generated by machine learning (e.g., k-nearest neighbor algorithm, support vector machine, neural network, etc.) using as input EMG data of a relaxed state and a state in which each of the five fingers is tense. That is, the force state identification unit 52 can input EMG data supplied from the communication unit 41 to the recognition model and output a recognition result indicating the force state of the user's fingers recognized from the EMG data. Note that when the force state identification unit 52 identifies the force state of the user's fingers by using a recognition model, the recognition model may be combined with the recognition model used by the hand posture identification unit 51 to form a single recognition model.

[0036] The initial action detection unit 53 detects that the user has performed an initial action, based on the user's hand shape identified by the hand shape identification unit 51 and the force state of the user's fingers identified by the force state identification unit 52. For example, the user performs the initial action when he or she wishes to start an operation on the information processing terminal 13 by inputting with strained fingers.

[0037] Here, the initial action is a motion pattern based on a gesture of applying pressure to the fingers in a combination that is not normally performed when the user grasps the real object 21, and a motion pattern that can be detected based on the myoelectric potential measured by the myoelectric potential measuring device 12 is used. The motion pattern of the initial action is registered in advance in the registration data of the storage unit 42, and the initial action can be registered in any motion pattern. For example, the registration data may be registered so that the initial action differs depending on the real object 21 to be grasped. Note that any graspable object in the real world can be used as the real object 21; for example, the information processing terminal 13 or a part of the user's own body may be used as the real object 21.

[0038] An example of an initial motion is shown in Fig. 6, and the numbers shown indicate the order in which the fingers are pressed. For example, as shown in Fig. 6A, a motion form in which the index finger, middle finger, ring finger, little finger, and thumb are pressed in order with the fingers gripping a rod-shaped real object 21A can be used as the initial motion. Also, as shown in Fig. 6B, a motion form in which the ring finger and little finger are pressed in order with the fingers pinching a small rectangular parallelepiped real object 21B with the thumb, index finger, and middle finger without applying force to the real object 21B can be used as the initial motion.

[0039] Other examples of initial movements that may be used include a movement in which the index finger and little finger are pressed simultaneously, a movement in which the thumb is pressed three times in succession within a short period of time, a movement in which the index finger and thumb are pressed continuously for five seconds, a movement in which the fingers are pressed in turn in one-second intervals, and a movement in which two stages of force are applied, with weak force followed by strong force.

[0040] For example, the initial movement is performed with the primary purpose of triggering an operation input to the information processing terminal 13. This prevents operation input contrary to the user's intention and explicitly indicates the start of operation input to the information processing terminal 13. Furthermore, in the terminal operation system 11, after the initial movement is detected, myoelectric potential data for identifying the user's hand shape and finger force state is acquired at a high sampling rate, thereby reducing power consumption. For example, in the terminal operation system 11, it is preferable to adopt, as the initial movement form, a movement in which the fingers are tense for a relatively long period of time so that myoelectric potential data can be acquired even at a low sampling rate, so that myoelectric potential data can be acquired at a low sampling rate for detecting the initial movement.

[0041] The second purpose of the initial operation is to use it as a calibration (initialization process) for setting a threshold value used to detect the force state of the fingers by the operation execution unit 55. For example, the maximum values ​​of the myoelectric potential data obtained in the initial operation when each finger is pressed can be stored, and an arbitrary ratio of these maximum values ​​can be set as the threshold value used by the operation execution unit 55.

[0042] For example, the left side of Fig. 7 shows an example of myoelectric potential data obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 when the user is in a stationary state during the initial movement, and this data is recorded as myoelectric potential data indicating the finger shape during the initial movement. The right side of Fig. 7 shows an example of myoelectric potential data obtained from the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 when the user tenses the fingers for the first time during the initial movement from this stationary state. The maximum value of the myoelectric potential data obtained when the user tenses the fingers for the first time is stored, and an arbitrary ratio of this maximum value, as indicated by the dashed dotted line, can be set as a threshold value used by the operation execution unit 55 to detect a force state in which the fingers are tense.

[0043] The terminal operation system 11 may identify the real object 21 being held or supported by the user from the motion pattern of the initial motion. Different initial motions may be associated with different real objects 21 that are held or supported in the same motion pattern. The terminal operation system 11 may also determine the frame rate for measuring the myoelectric potential from the result of the initial motion. For example, the frame rate may be set to be lowered when the myoelectric potential fluctuates greatly and is easy to identify.

[0044] When the user's hand shape identified by the hand shape identification unit 51 is registered in the registration data stored in the storage unit 42, the operation mode selection unit 54 selects the operation mode associated with the hand shape. Furthermore, when the user's hand shape identified by the hand shape identification unit 51 is not registered in the registration data stored in the storage unit 42, the operation mode selection unit 54 selects the standard mode as the operation mode. Note that, in addition to having the operation mode selection unit 54 select an operation mode, the user may directly operate the information processing terminal 13 to select a desired operation mode. Furthermore, the operation mode associated with the user's hand shape and registered in the registration data in the storage unit 42 can be set arbitrarily, and the operation content for each operation mode can also be set arbitrarily.

[0045] In the terminal operation system 11, even if the user does not perform an initial operation, if it is inferred from the electromyography data that the user's hand shape matches a pre-registered hand shape at the time of the initial operation, the operation mode selection unit 54 can select an operation mode corresponding to that hand shape.

[0046] When the operation execution unit 55 detects that the force state of the user's fingers identified by the force state identification unit 52 after the detection of the initial operation corresponds to the force state of the operation content in the operation mode selected by the operation mode selection unit 54, the operation execution unit 55 executes processing according to the operation content corresponding to the force state. For example, the operation execution unit 55 can detect the force state of the user's fingers based on a binary value (0, 1) indicating whether or not the strain of each finger exceeds a threshold. Alternatively, the operation execution unit 55 may detect the force state of the user's fingers based on a linear output (0 to 100%) of the strain of each finger.

[0047] The hand shape, operation mode, and operation content will be described with reference to FIGS. 8 and 9. FIG.

[0048] 8 , when the user's fingering position is such that the right hand is open and all fingertips are in contact with an object, the operation mode selection unit 54 selects the mouse operation mode as the operation mode. In the mouse operation mode, the operation execution unit 55 executes a process according to an operation content corresponding to a left button click depending on the force state of the index finger, and executes a process according to an operation content corresponding to a double click depending on the force state of the index finger. Similarly, in the mouse operation mode, the operation execution unit 55 executes a process according to an operation content corresponding to a right button click depending on the force state of the middle finger, executes a process according to an operation content corresponding to a back button click depending on the force state of the thumb, and executes a process according to an operation content corresponding to a forward button click depending on the force state of the ring finger.

[0049] When the user's fingering position is such that the left hand is open and all fingertips other than the little finger are in contact with an object, the operation mode selection unit 54 selects the cross key operation mode. In the cross key operation mode, the operation execution unit 55 executes a process according to an operation content corresponding to the upward direction depending on the force state of the middle finger, and executes a process according to an operation content corresponding to the leftward direction depending on the force state of the ring finger. Similarly, in the cross key operation mode, the operation execution unit 55 executes a process according to an operation content corresponding to the rightward direction depending on the force state of the index finger, and executes a process according to an operation content corresponding to the downward direction depending on the force state of the thumb.

[0050] When the user's finger shape is gripping the steering wheel, the operation mode selection unit 54 selects the map operation mode as the operation mode. In the map operation mode, the operation execution unit 55 executes processing according to the operation content to enlarge the map in accordance with the force state changing from the tension of the index finger to the tension of the middle finger, and executes processing according to the operation content to reduce the map in accordance with the force state changing from the tension of the middle finger to the tension of the index finger. Similarly, in the map operation mode, the operation execution unit 55 executes processing according to the operation content to play the next guidance in accordance with the force state of the thumb continuously tensioning.

[0051] When the user's fingering position is such that the earphones are pinched with the index finger and thumb, the operation mode selection unit 54 selects the volume operation mode as the operation mode. In the volume operation mode, the operation execution unit 55 executes processing according to the operation content to increase the volume depending on the force state of the index finger, and executes processing according to the operation content to decrease the volume depending on the force state of the thumb. Similarly, in the volume operation mode, the operation execution unit 55 executes processing according to the operation content to stop or start the sound depending on the force state of all fingers.

[0052] 9A, when the user's hand shape is not registered in the registration data stored in the storage unit 42, the operation mode selection unit 54 selects the standard mode as the operation mode. In the standard mode, the operation execution unit 55 executes processing according to the operation content corresponding to moving forward depending on the force state of the index finger, executes processing according to the operation content corresponding to moving back depending on the force state of the middle finger, and executes processing according to the operation content corresponding to deciding depending on the force state of the thumb.

[0053] When the user's finger shape is an OK sign (a hand sign in which the thumb and index finger form a circle) as shown in B of Fig. 9, the operation mode selection unit 54 selects the Yes-No mode as the operation mode. In the Yes-No mode, the operation execution unit 55 executes a process according to the operation content corresponding to Yes depending on the force state of the index finger and thumb, and executes a process according to the operation content corresponding to No depending on the force state of the middle finger and thumb.

[0054] The hand shapes, operation modes, and operation contents described with reference to FIGS. 8 and 9 are merely examples, and other hand shapes, operation modes, and operation contents may be used.

[0055] As described above, the terminal operation system 11 can cause the user to perform an initial operation using the strength of the user's five fingers while holding or supporting any real object 21, thereby enabling the user to start inputting an operation to the information processing terminal 13. Then, based on the shape of the user's fingers and the force state of the fingers, the terminal operation system 11 can execute a process according to the corresponding operation content, for example, a different process depending on the force exerted by each of the five fingers by the user.

[0056] Furthermore, the terminal operation system 11 can use any graspable real object 21 in the real world to input operations to the information processing terminal 13. For example, even if different real objects 21 are used, the terminal operation system 11 can execute processing according to the same operation content as long as the hand shape and the combination of fingers to be pressed are the same. In other words, the terminal operation system 11 can operate the information processing terminal 13 without being limited to using known real objects 21 such as a pen-shaped, elongated object. In this way, the terminal operation system 11 can achieve greater operability with a higher degree of freedom by using a variety of arbitrary real objects 21.

[0057] <Processing Examples of EMG Measurement Processing and Operation Execution Processing> The EMG measurement processing and operation execution processing executed in the terminal operation system 11 will be described with reference to the flowcharts shown in FIGS.

[0058] The myoelectric potential measurement process executed by the myoelectric potential measuring device 12 will be described with reference to the flowchart shown in FIG.

[0059] In step S11, the communication unit 34 determines whether or not it has received a signal instructing the start of communication transmitted from the communication unit 41 of the information processing terminal 13, and waits until it determines that it has received the signal instructing the start of communication. If the communication unit 34 determines that it has received the signal instructing the start of communication, the process proceeds to step S12.

[0060] In step S12, the signal acquiring unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 and supplies them to the signal processing unit 33.

[0061] In step S13, the signal processing unit 33 acquires myoelectric potential data obtained as a result of performing signal processing such as noise removal processing and Fourier transform processing on the myoelectric potential signal supplied in step S12, and supplies the data to the communication unit 34.

[0062] In step S14, the communication unit 34 transmits the myoelectric potential data supplied in step S13 to the information processing terminal 13.

[0063] In step S15, the communication unit 34 determines whether or not it has received a signal instructing it to end communication transmitted from the communication unit 41 of the information processing terminal 13. If the communication unit 34 determines that it has not received a signal instructing it to end communication, the process returns to step S12, and the same process is repeated thereafter. On the other hand, if the communication unit 34 determines that it has received a signal instructing it to end communication, the process proceeds to step S16.

[0064] In step S16, various processes for ending the operation of the myoelectric potential measuring device 12 are performed, and then the myoelectric potential measuring process is ended.

[0065] The operation execution process executed in the information processing terminal 13 will be described with reference to the flowchart shown in FIG.

[0066] In step S21, the communication unit 41 transmits a signal to the communication unit 34 of the myoelectric potential measuring device 12 to instruct it to start communication.

[0067] In step S22, the communication unit 41 receives the myoelectric potential data transmitted from the communication unit 34 in step S14 of FIG. 10, and supplies the data to the hand posture specifying unit 51 and the force state specifying unit 52.

[0068] In step S23, the hand posture identification unit 51 identifies the user's hand posture based on the myoelectric potential data supplied in step S22, and supplies the recognition result indicating the user's hand posture to the initial action detection unit 53 and the operation mode selection unit 54.

[0069] In step S24, the force state identification unit 52 identifies the force state of the user's finger based on the EMG data supplied in step S22, and supplies the recognition result indicating the force state of the user's finger to the initial action detection unit 53 and the operation execution unit 55.

[0070] In step S25, the initial movement detection unit 53 determines whether or not an initial movement by the user has been detected, based on the user's hand and finger shape identified in step S23 and the force state of the user's fingers identified in step S24.

[0071] If the initial action detection unit 53 determines in step S25 that an initial action by the user has been detected, the process proceeds to step S26.

[0072] In step S26, the operation mode selection unit 54 determines whether the user's hand shape identified in step S23 is registered in the registration data stored in the storage unit 42.

[0073] If the operation mode selection unit 54 determines in step S26 that the user's hand shape is registered in the registration data, the process proceeds to step S27. In step S27, the operation mode selection unit 54 selects the operation mode associated in the registration data with the user's hand shape identified in step S23, and the process proceeds to step S30.

[0074] On the other hand, if the operation mode selection unit 54 determines in step S26 that the user's hand shape is not registered in the registration data (is unregistered), the process proceeds to step S28. In step S28, the operation mode selection unit 54 selects the standard mode as the operation mode, and the process proceeds to step S30.

[0075] On the other hand, if the initial action detection unit 53 determines in step S25 that an initial action by the user has not been detected, the process proceeds to step S29.

[0076] In step S29, the operation mode selection unit 54 determines whether the user's hand shape identified in step S23 matches the hand shape at the time of the initial operation. If the operation mode selection unit 54 determines that the user's hand shape does not match the hand shape at the time of the initial operation, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the operation mode selection unit 54 determines that the user's hand shape matches the hand shape at the time of the initial operation, an operation mode corresponding to the hand shape is selected, and the process proceeds to step S30.

[0077] In step S30, the operation execution unit 55 determines whether or not a force state corresponding to the force state of the operation content in the selected operation mode has been detected from the force state of the user's finger identified by the force state identification unit 52 after the detection of the initial operation.

[0078] If the operation execution unit 55 determines in step S30 that a force state corresponding to the force state of the operation content in the selected operation mode has not been detected, the process returns to step S22, and the same processes are repeated thereafter. On the other hand, if the operation execution unit 55 determines in step S30 that a force state corresponding to the force state of the operation content in the selected operation mode has been detected, the process proceeds to step S31.

[0079] In step S31, the operation execution unit 55 executes processing according to the operation content corresponding to the force state of the user's finger detected in step S30.

[0080] In step S32, the calculation unit 43 determines whether or not to end the operation execution process.

[0081] If the calculation unit 43 determines in step S32 not to end the operation execution process, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the calculation unit 43 determines to end the operation execution process, the process proceeds to step S33.

[0082] In step S33, the communication unit 41 transmits a signal to the communication unit 34 of the electromyography measuring device 12 to instruct the communication unit 34 to end communication, and then the operation execution process ends.

[0083] As described above, the terminal operation system 11 can utilize various arbitrary real objects 21 to achieve operability with a higher degree of freedom.

[0084] <Examples of Use and Applications of the Terminal Operation System> Examples of use and applications of the terminal operation system 11 will be described with reference to FIGS.

[0085] FIG. 12 is a diagram illustrating an example of a use case in which a smart watch is used as the terminal operation system 11.

[0086] For example, when a smartwatch is used as the terminal operation system 11, in the example shown in Fig. 12, a myoelectric potential measuring device 12 is incorporated into the band of the smartwatch, and an information processing terminal 13 provided on the main body of the smartwatch executes the above-mentioned operation execution process. Also, in the example shown in Fig. 12, a bicycle handlebar is used as the real object 21C.

[0087] When the user's fingers are gripping the steering wheel, the map operation mode is selected as the operation mode, as shown in A of FIG.

[0088] When a user activates a map app on the smartwatch, which is the terminal operation system 11, to travel to a destination by bicycle, a map is displayed on the display unit of the terminal operation system 11, and navigation to the destination begins, as shown in B of Fig. 12. Then, the user performs an initial motion, for example, by simultaneously pressing the index finger and little finger, and then simultaneously pressing the middle finger and ring finger, thereby starting operation input to the terminal operation system 11 and pedaling the bicycle.

[0089] For example, when the user wants to know the timing of the next right turn, if the user applies pressure to the thumb continuously, a navigation message "turn right in 450 m" will be displayed on the display unit of the terminal operation system 11 in accordance with the operation content corresponding to the pressure applied, as shown in C of FIG. 12.

[0090] Furthermore, when a user wants to check a destination on a map but the destination cannot be displayed at the current map scale, by first pressing the index finger and then pressing the middle finger, the map displayed on the display unit of the terminal operation system 11 is enlarged in accordance with the operation content corresponding to the state of pressure, as shown in D of Fig. 12. The map can be enlarged so that the magnification changes linearly according to the strength of the pressure applied to the fingers at this time.

[0091] FIG. 13 is a diagram illustrating an example of a usage example in which a user uses AR glasses as the information processing terminal 13 and a spatula as the real object 21D when cooking.

[0092] For example, a user wears AR glasses, which are the information processing terminal 13, on their head and plays a recipe video, and is cooking while holding a spatula in their right hand, which is wearing the myoelectric potential measuring device 12, and a frying pan in their left hand. At this time, there may be a situation where the user wants to return to the recipe video displaying a list of ingredients to check the next ingredient to be added, but cannot take their hands off the screen to avoid burning the food. In such a situation, the user performs an initial movement by applying pressure to the index finger, middle finger, and thumb in the right hand holding the spatula.

[0093] At this time, if the hand shape holding the spatula is not registered in the registration data of the storage unit 42, the standard mode (A in FIG. 13) is selected as the operation mode. Then, the AR glasses, which are the information processing terminal 13, display operation guides indicated by dashed lines superimposed on each finger according to the operation content of the standard mode, as shown in B in FIG. 13. That is, an operation guide representing the operation content "forward" is displayed superimposed on the index finger, an operation guide representing the operation content "back" is displayed superimposed on the middle finger, and an operation guide representing the operation content "play" is displayed superimposed on the thumb.

[0094] Therefore, when the user applies pressure to the middle finger of their right hand while holding the spatula, the playback position of the recipe video is rewound by 30 seconds in accordance with the operation content corresponding to the pressure applied, and the user continues to apply pressure to the middle finger until the playback position of the recipe video returns to the scene the user wants to see. Then, when the user applies pressure to their thumb at the timing when the playback position of the recipe video returns to the scene the user wants to see, the recipe video is played back according to the operation content corresponding to the pressure applied.

[0095] In addition, in the terminal operation system 11, when the user has successfully executed an operation by applying pressure to the finger, the AR glasses, which are the information processing terminal 13, may notify the user by video, audio, or vibration. For example, as shown in C of Fig. 13, when the user has successfully executed the operation content "back" by applying pressure to the middle finger, the notification may be made by making a small change to the display of the operation guide. Also, for example, the notification may be made by using a different color for each finger or by using a different musical scale for each finger.

[0096] Furthermore, since it is expected that as time passes from the start of operation of the terminal operation system 11, the user will become tired and the strength of the force applied to the fingers will decrease, the threshold for the strength of the force required for operation may be lowered as time passes from the start of operation.

[0097] For example, as shown in A of Fig. 14, if a certain time has not elapsed since the start of the operation, the operation guide superimposed on the user's fingers gripping the rod-shaped real object 21E is displayed in a normal size. After that, as shown in B of Fig. 14, if a certain time has elapsed since the start of the operation, the operation guide superimposed on the user's fingers gripping the rod-shaped real object 21E is displayed in a smaller size than normal to indicate that the threshold force required for the operation has decreased. In this case, conversely, the operation guide may be displayed in a larger size than normal to encourage the user to exert more force when operating.

[0098] An application example will be described in which a camera is mounted on the information processing terminal 13 and the real object 21 can be detected by the camera.

[0099] For example, when the information processing terminal 13 identifies a known real object 21 with a camera, it can select an operation mode corresponding to the known real object 21 without performing any initial operation. On the other hand, when the information processing terminal 13 identifies an unknown real object 21 with a camera, it can select a standard mode without performing any initial operation. Furthermore, when the information processing terminal 13 is capable of superimposing display using AR, it may display a hand icon superimposed on the real object 21 identified with the camera and display operation details that can be performed using the real object 21. Furthermore, when the information processing terminal 13 is capable of detecting the user's hand with a camera, it can recognize the hand and finger shape not only using myoelectric potential data but also using an image of the hand captured by the camera.

[0100] Furthermore, the information processing terminal 13 may use a camera to identify the relative position of the finger that the user is pressing, and change the processing according to the position. For example, as shown in Fig. 15, when an operation is performed on a keyboard, it is possible to change the processing to be performed according to the position even when the index finger is pressed, such that when the index finger is pressed at a certain position, it is assumed that an operation to input "J" has been performed, and when the index finger is pressed at a position that is moved parallel to the position about 2 cm to the right from that position, it is assumed that an operation to input "K" has been performed.

[0101] Furthermore, as an application example, the terminal operation system 11 may select a specific operation mode and accept operations for the information processing terminal 13 based on the myoelectric potential data for a certain period of time immediately after some action occurs on the information processing terminal 13 side, even if no initial operation is performed assuming that an operation input will be made. Then, when acceptance of operations has started in the specific operation mode, the sampling frequency for measuring the myoelectric potential data may be increased.

[0102] Furthermore, in the terminal operation system 11, if the information processing terminal 13 is a smartphone, when there is a voice message (such as an incoming phone call), the terminal may be configured to accept operations on the information processing terminal 13 based on the electromyogram data for a certain period of time thereafter. At this time, it is expected that an operation to answer or reject the call will be performed based on the electromyogram data, so a Yes-No mode (see B in FIG. 9 ) can be selected.

[0103] Furthermore, in the terminal operation system 11, in preparation for an operation unintended by the user, an operation (cancel operation) for canceling the previous operation input can be set to be available in all operation modes. For example, the cancel operation can be a force state in which all fingers are spread apart or a force state in which all fingers are tensed. Furthermore, in the terminal operation system 11, if a cancel operation is performed a certain number of times or more, the operation input to be canceled may be considered to be an erroneous detection and processing corresponding to the operation input may not be executed. The terminal operation system 11 can also learn from the results of the cancel operation and correct the operation mode.

[0104] Furthermore, the terminal operation system 11 can be used for sports analysis. For example, when gripping a golf club (real object 21), the terminal operation system 11 can save the way each finger applies force rather than explicit operation input, analyze it as a golfing motion, and provide feedback such as "Relax your thumb." Similarly, the terminal operation system 11 can measure the user's level of concentration on driving from the way the steering wheel (real object 21) is held while driving.

[0105] Furthermore, the terminal operation system 11 can identify an individual based on the posture of the person holding the real object 21 and the force applied when holding the real object 21. For example, the processing may be performed by referring to the registered data for each individual.

[0106] In addition, if the terminal operation system 11 can identify the user's hand shape and finger force state, it may use measurement data obtained by detecting changes in muscle sounds, ultrasound, electromagnetic waves, capacitance, etc. using sensors, in addition to using myoelectric potential data.

[0107] Furthermore, the terminal operation system 11 may be configured such that the storage unit 42 and some blocks of the calculation unit 43 are provided on a remote server, and the server and the information processing terminal 13 are constantly connected via a network. In this configuration, the information processing terminal 13 needs to have a function for synchronizing with the remote server via wired or wireless communication. In addition, in this configuration, the motion patterns of initial operations, the correspondence between the user's finger shapes and operation modes, the operation contents for each operation mode, etc. are stored on the remote server, so that even if different information processing terminals 13 are used, the same operation input can be performed as long as the same real object 21 is used.

[0108] <Example of Computer Configuration> Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0109] FIG. 16 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0110] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) 104 are interconnected by a bus 105. An input / output interface 106 is further connected to the bus 105, and the input / output interface 106 is connected to the outside.

[0111] In a computer configured as described above, the CPU 101 performs the above-described series of processes by loading programs stored in, for example, the ROM 102 and EEPROM 104 into the RAM 103 via the bus 105 and executing the programs. In addition, the programs executed by the computer (CPU 101) can be written in advance in the ROM 102, or can be installed or updated in the EEPROM 104 from outside via the input / output interface 106.

[0112] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0113] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0114] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0115] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0116] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0117] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0118] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0119] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0120] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0121] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) An information processing device comprising: an acquisition unit that acquires measurement data corresponding to the movement of a user's hand and fingers while gripping or supporting an arbitrary real object; and a calculation unit that estimates the user's hand shape and force state based on the measurement data and executes processing corresponding to the user's operation. (2) The information processing device described in (1) above, wherein the calculation unit has: a hand shape identification unit that identifies a hand shape representing the shape of the user's hand and fingers based on the measurement data; and a force state identification unit that identifies which of the user's five fingers is tensing and a force state representing the strength of the tensing of that finger based on the measurement data. (3) The information processing device described in (2) above, wherein the calculation unit further has an initial action detection unit that detects that the user has performed an initial action based on the user's hand shape identified by the hand shape identification unit and the force state of the user's fingers identified by the force state identification unit. (4) The information processing device according to (3) above, wherein the initial action uses a gesture in which the user, who is gripping or supporting the real object, applies pressure to his or her fingers in a predetermined combination. (5) The information processing device according to (3) above, wherein the calculation unit further has an operation mode selection unit that, when the initial action detection unit detects that the user has performed an initial action, selects an operation mode associated with the user's hand shape identified by the hand shape identification unit. (6) The information processing device according to (5) above, wherein, when the calculation unit detects that the force state of the user's fingers identified by the force state identification unit corresponds to a force state of operation content in the operation mode selected by the operation mode selection unit, the calculation unit further has an operation execution unit that executes processing in accordance with operation content associated with the force state. (7) The information processing device according to (6) above, wherein the operation mode selection unit selects a map operation mode as the operation mode when the hand shape identification unit identifies that the user's hand shape is gripping a steering wheel.(8) The information processing device described in (7) above, wherein, when the map operation mode is selected by the operation mode selection unit, the operation execution unit executes a process according to an operation content corresponding to "zoom in on the map" in accordance with a force state that changes from stiffening of the index finger to stiffening of the middle finger, executes a process according to an operation content corresponding to "zoom out on the map" in accordance with a force state that changes from stiffening of the middle finger to stiffening of the index finger, and executes a process according to an operation content corresponding to "play back next guidance" in accordance with a force state that keeps stiffening the thumb. (9) The information processing device described in (6) above, wherein the operation mode selection unit selects a standard mode as the operation mode when the user's hand shape identified by the hand shape identification unit is unregistered. (10) The information processing device according to any of (5) to (10) above, wherein, when the standard mode is selected by the operation mode selection unit, the operation execution unit performs a process according to an operation content corresponding to "advance" according to a force state of a tensed index finger, a process according to an operation content corresponding to "return" according to a force state of a tensed middle finger, and a process according to an operation content corresponding to "confirm" according to a force state of a tensed thumb. (11) The information processing device according to any of (5) to (10) above, wherein, even if the user does not perform an initial action, when the hand shape identified by the hand shape identification unit matches the hand shape at the time of the initial action, the operation mode selection unit selects the operation mode corresponding to the hand shape. (12) The information processing device according to any of (1) to (11) above, wherein the measurement data is myoelectric potential data obtained as a result of measuring myoelectric potential of the user's arm. (13) The information processing device according to (12), wherein the information processing device is a smartwatch having a band incorporating a myoelectricity measuring device that outputs the myoelectricity data, and at least the calculation unit is provided in a main body of the smartwatch. (14) An information processing method, including: by an information processing device, acquiring measurement data corresponding to the movement of a user's hand and fingers while gripping or supporting an arbitrary real object, estimating the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and executing processing corresponding to an operation of the user.(15) A program for causing a computer of an information processing device to execute information processing including: acquiring measurement data according to the movement of a user's hand and fingers while grasping or supporting any real object; and estimating the shape of the user's fingers and the force state of the hand based on the measurement data, and executing processing according to the user's operation.

[0122] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0123] REFERENCE SIGNS LIST 11 Terminal operation system, 12 Myoelectric potential measuring device, 13 Information processing terminal, 21 Real object, 31 Myoelectric potential sensor, 32 Signal acquisition unit, 33 Signal processing unit, 34 Communication unit, 41 Communication unit, 42 Memory unit, 43 Calculation unit, 51 Hand shape identification unit, 52 Force state identification unit, 53 Initial operation detection unit, 54 Operation mode selection unit, 55 Operation execution unit

Claims

1. An information processing device comprising: an acquisition unit that acquires measurement data according to the movement of a user's hand and fingers while grasping or supporting any real object; and a calculation unit that estimates the shape of the user's fingers and the force state of the hand based on the measurement data and executes processing according to the user's operation.

2. The information processing device according to claim 1, wherein the calculation unit has: a hand shape identification unit that identifies a hand shape representing the shape of the user's hand and fingers based on the measurement data; and a force state identification unit that identifies, based on the measurement data, which of the user's five fingers is tensing and a force state representing the strength of the tensing of that finger.

3. The information processing device according to claim 2, wherein the calculation unit further includes an initial movement detection unit that detects that the user has performed an initial movement based on the user's hand shape identified by the hand shape identification unit and the force state of the user's fingers identified by the force state identification unit.

4. The information processing device according to claim 3, wherein the initial motion is a motion form based on a gesture in which the user, who is holding or supporting the given real object, applies pressure to the fingers in a predetermined combination.

5. The information processing device according to claim 3, wherein the calculation unit further has an operation mode selection unit that selects an operation mode associated with the user's hand shape identified by the hand shape identification unit when the initial movement detection unit detects that the user has performed an initial movement.

6. The information processing device according to claim 5, wherein the calculation unit further has an operation execution unit that, when it detects that the force state of the user's finger identified by the force state identification unit corresponds to the force state of the operation content in the operation mode selected by the operation mode selection unit, executes processing in accordance with the operation content corresponding to the force state.

7. The information processing device according to claim 6, wherein the operation mode selection unit selects the map operation mode as the operation mode when the hand posture identification unit identifies that the user's hand posture is gripping a steering wheel.

8. The information processing device according to claim 7, wherein, when the map operation mode is selected by the operation mode selection unit, the operation execution unit executes processing according to an operation content corresponding to "zooming in on the map" in accordance with a force state that changes from tension in the index finger to tension in the middle finger, executes processing according to an operation content corresponding to "zooming out on the map" in accordance with a force state that changes from tension in the middle finger to tension in the index finger, and executes processing according to an operation content corresponding to "playing back the next guidance" in accordance with a force state in which the thumb is continuously tense.

9. The information processing device according to claim 6, wherein the operation mode selection unit selects the standard mode as the operation mode when the user's hand posture identified by the hand posture identification unit is unregistered.

10. The information processing device according to claim 9, wherein, when the standard mode is selected by the operation mode selection unit, the operation execution unit executes processing according to an operation content corresponding to "forward" according to the force state of the index finger, executes processing according to an operation content corresponding to "back" according to the force state of the middle finger, and executes processing according to an operation content corresponding to "decide" according to the force state of the thumb.

11. The information processing device according to claim 5, wherein the operation mode selection unit selects an operation mode corresponding to the user's hand shape when the user's hand shape identified by the hand shape identification unit matches the hand shape at the time of the initial action, even if the user does not perform an initial action.

12. The information processing device according to claim 1, wherein the measurement data is myoelectric potential data obtained as a result of measuring the myoelectric potential of the user's arm.

13. The information processing device according to claim 12, wherein the information processing device is a smartwatch having a myoelectricity measuring device built into a band that outputs the myoelectricity data, and at least the calculation unit is provided in the main body of the smartwatch.

14. An information processing method including: an information processing device acquiring measurement data corresponding to the movement of a user's hand and fingers while grasping or supporting an arbitrary real object; estimating the shape of the user's fingers and the force state of the hand based on the measurement data; and executing processing corresponding to the user's operation.

15. A program for causing a computer of an information processing device to perform information processing including: acquiring measurement data according to the movement of a user's hand and fingers while grasping or supporting any real object; estimating the shape of the user's fingers and the force state of the hand based on the measurement data; and performing processing according to the user's operation.

Images