Information processing apparatus, information processing method, and program

Abstract

The present disclosure relates to an information processing apparatus, an information processing method, and a program capable of achieving higher degrees of freedom of operability. An information processing apparatus according to the present disclosure estimates a user's finger and hand shape and a force state of the user's hand based on electromyogram data corresponding to movement of the user's hand and fingers in a state of gripping or supporting a given real object, and performs processing corresponding to an operation performed by the user. Furthermore, the information processing apparatus detects that the user has performed an initial operation based on the user's finger and hand shape and the force state of the user's fingers. For example, the present technology can be applied to a terminal operating system using a given real object.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods and procedures, and particularly to information processing apparatus, information processing methods and procedures that enable more flexible operation. Background Technology

[0002] Traditionally, technologies have been developed for measuring the electromyography (EMG) of a user's arm and for operating information processing terminals based on the movement of the user's hand and fingers.

[0003] For example, Patent Document 1 discloses a motion recognition method using a grasping object, wherein the user's wrist state is estimated by the writing motion of grasping the object, the joint motion of the user's body parts associated with the wrist caused by the writing motion is estimated, and the state of the grasping object is estimated by the wrist state and the joint motion.

[0004] Citation List

[0005] Patent documents

[0006] PTL 1: JP 2015-001978 A Summary of the Invention

[0007] Technical issues

[0008] As disclosed in Patent Document 1 above, in addition to recognizing the content written using an object with a slender, pen-like shape, it is also necessary to achieve highly flexible operability by enabling the use of various arbitrary objects.

[0009] This disclosure was made in light of these circumstances and is intended to enable greater operational flexibility.

[0010] Solution to the problem

[0011] The information processing apparatus according to an aspect of this disclosure includes: an acquisition unit configured to acquire measurement data corresponding to the movement of a user's hand and fingers in a state of grasping or supporting any real object; and a processing unit configured to estimate the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and to perform processing according to the user's operation.

[0012] Information processing methods or procedures according to aspects of this disclosure include: acquiring measurement data corresponding to the movement of a user's hand and fingers in a state of grasping or supporting any real object, estimating the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and performing processing according to the user's operation.

[0013] According to aspects of this disclosure, measurement data corresponding to the movement of a user's hand and fingers in a state of grasping or supporting any real object is acquired, and the user's hand and finger shape and hand force state are estimated based on the measurement data, and processing is performed according to the user's operation. Attached Figure Description

[0014] [ Figure 1 [ ] is a block diagram illustrating a configuration example of an implementation of a terminal operating system using this technology.

[0015] [ Figure 2 [Illustration 1] is a block diagram showing an example configuration of a myoelectric potential measurement device and an information processing terminal.

[0016] [ Figure 3 [Illustration 1] is a diagram used to describe a recognition model for identifying the shape of a user's hand and fingers.

[0017] [ Figure 4 [ ] is a graph used to describe the identification of force states based on thresholds.

[0018] [ Figure 5 ] is a diagram used to describe the identification of force states using an identification model.

[0019] [ Figure 6 ] is a graph used to describe the detection of the initial action.

[0020] [ Figure 7 ] is a diagram used to describe the calibration used to set the threshold.

[0021] [ Figure 8 A diagram is used to describe the shape of the hand and fingers, their operating patterns, and the content of their operations.

[0022] [ Figure 9 A diagram is used to describe the shape of the hand and fingers, their operating patterns, and the content of their operations.

[0023] [ Figure 10 [ ] is a flowchart used to describe the processing of myoelectric potential measurements.

[0024] [ Figure 11 [] is a flowchart used to describe the execution and processing of an operation.

[0025] [ Figure 12 [Illustration] is a diagram used to illustrate an example of using a smartwatch.

[0026] [ Figure 13 [] is a diagram used to describe an example of how a user cooks.

[0027] [ Figure 14 [ ] is an example diagram used to illustrate how the display size of the user guide changes over time.

[0028] [ Figure 15 [] is a diagram used to illustrate examples of keyboard operations.

[0029] [ Figure 16 [ ] is a block diagram illustrating a configuration example of an implementation of the present technology in a computer. Detailed Implementation

[0030] The following will describe in detail, with reference to the accompanying drawings, specific implementation methods of this technology.

[0031] Terminal operating system configuration example

[0032] Figure 1 This is a diagram illustrating a configuration example of an implementation of a terminal operating system using this technology.

[0033] like Figure 1 As shown, the terminal operating system 11 is configured to include a myoelectric potential measurement device 12 and an information processing terminal 13.

[0034] The electromyography (EMG) measuring device 12 measures the electromyographic activity (EMG) of the arm of a user using the terminal operating system 11, communicates with the information processing terminal 13, and sends the EMG data obtained as a result of measuring the user's EMG. Although in Figure 1 The image shows a myoelectric potential measuring device 12 with a ring shape that is worn on the user's right arm, but the myoelectric potential measuring device 12 can also be worn on the user's left or right arm at any position from the shoulder to the wrist.

[0035] For example, the electromyography (EMG) measuring device 12 is configured to include a plurality of EMG sensors in close contact with the skin surface, and each EMG sensor includes at least three electrodes: a positive electrode, a negative electrode, and a reference electrode. Note that the electrodes included in the EMG sensors can be configured such that the reference electrode is positioned on a straight line connecting the positive and negative electrodes, or can be configured such that the reference electrode is positioned outside of a straight line. The EMG measuring device 12 can employ a configuration in which the plurality of EMG sensors are arranged in a grid pattern.

[0036] Information processing terminal 13 is an electronic device operated by a user using terminal operating system 11. Figure 1 In the example shown, an augmented reality (AR) or virtual reality (VR) head-mounted display is illustrated as an example of information processing terminal 13, but smartphones, smartwatches, etc., can also be used. Information processing terminal 13 communicates with electromyography (EMG) measuring device 12, receives EMG data sent from EMG measuring device 12, and performs processing based on the EMG data according to the user's operation on information processing terminal 13.

[0037] For example, when a user grasps or supports any real object 21 and performs an initial action by applying force with multiple fingers, the electromyography (EMG) measuring device 12 measures the EMG of the user's arm according to the manner in which each finger applies force, and the information processing terminal 13 detects that the initial action has been performed based on the EMG data. Then, the information processing terminal 13 can identify the shape of the user's hand and fingers (the shape of the hand and fingers) and the force state of the user's fingers (e.g., the force applied by the five fingers and the strength of the force of the corresponding fingers) based on the EMG data, and can determine the content of the operation performed by the user on the information processing terminal 13 based on the estimation results.

[0038] Therefore, a user wearing the electromyography (EMG) measuring device 12 can perform various operations on the information processing terminal 13 by grasping or supporting any real object 21 with the predetermined shape of their hands and fingers and applying force with the desired fingers.

[0039] Note that in Figure 1 In the terminal operating system 11 shown, the information processing terminal 13 is the user's operating object. This information processing terminal 13 processes the recognition of the user's hand and finger shape and the force state of the fingers, and determines the operation content to be performed by the user. However, the user's operating object can also be an electronic device other than the information processing terminal 13 that performs such processing. That is, the information processing terminal 13 can notify another electronic device of the operation content, and cause the processing according to the user's operation to be executed.

[0040] Figure 2 This is a block diagram showing an example configuration of the electromyography device 12 and the information processing terminal 13.

[0041] like Figure 2 As shown, the electromyography (EMG) measuring device 12 is configured to include eight EMG sensors 31-1 to 31-8, a signal acquisition unit 32, a signal processing unit 33, and a communication unit 34, and the information processing terminal 13 is configured to include a communication unit 41, a storage unit 42, and a processing unit 43.

[0042] Myoelectric potential sensors 31-1 to 31-8 in such Figure 1 As shown, with the electromyographic (EMG) measuring device 12 worn on the user's arm, it is positioned at predetermined intervals along the periphery of the user's arm, and outputs EMG signals (electrical signals obtained by measuring and amplifying the potential difference between the positive, negative, and reference electrodes) corresponding to the weak electric field generated in the muscles of the user's arm at the corresponding positions. Note that, without distinguishing between EMG sensors 31-1 to 31-8, EMG sensors 31-1 to 31-8 are hereinafter simply referred to as EMG sensor 31. Furthermore, although in Figure 2The example shown uses eight electromyographic sensors 31-1 to 31-8, but the number of electromyographic sensors 31 disposed on the electromyographic measuring device 12 may be eight or more, or may be eight or fewer.

[0043] The signal acquisition unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 and supplies the myoelectric potential signals to the signal processing unit 33.

[0044] The signal processing unit 33 performs signal processing (e.g., noise removal and Fourier transform processing) on ​​the myoelectric potential signal supplied from the signal acquisition unit 32, for example, by using circuits or digital processing such as operational amplifiers and high-pass filters. Then, the signal processing unit 33 supplies the myoelectric potential data (a waveform indicating the change of myoelectric potential according to the movement of the user's hand and fingers) obtained as a result of performing signal processing on the myoelectric potential signal to the communication unit 34.

[0045] The communication unit 34 communicates with the communication unit 41 of the information processing terminal 13 to send myoelectric potential data supplied from the signal processing unit 33.

[0046] The communication unit 41 communicates with the communication unit 34 of the electromyography (EMG) measuring device 12 to receive EMG data and supply the EMG data to the processing unit 43. Note that the communication unit 34 and the communication unit 41 can communicate using a wired cable, a wireless local area network (LAN), Bluetooth (registered trademark), etc.

[0047] Storage unit 42 is a memory or storage device that stores pre-registered data such as the action pattern of the initial action, the correspondence between the user's hand and finger shapes and the operation mode, and the operation content of each operation mode.

[0048] The arithmetic unit 43 is configured with a central processing unit (CPU), integrated circuits (ICs), etc., and performs calculations to estimate the shape of the user's hand and fingers and the force state of the hand based on the myoelectric data supplied from the communication unit 41 using machine learning models, recognition models, algorithms, etc. Then, the arithmetic unit 43 obtains the operation mode corresponding to the user's hand and finger shape from the registration data in the storage unit 42, and performs processing according to the operation content corresponding to the force state of the user's hand in the operation mode.

[0049] As shown in the figure, the arithmetic unit 43 is configured to include a hand and finger shape recognition unit 51, a force state recognition unit 52, an initial motion detection unit 53, an operation mode selection unit 54, and an operation execution unit 55.

[0050] The hand and finger shape recognition unit 51 recognizes the user's hand and finger shape based on the myoelectric data supplied from the communication unit 41, and supplies the recognition results indicating the user's hand and finger shape to the initial motion detection unit 53 and the operation mode selection unit 54.

[0051] For example, such as Figure 3 As shown, electromyographic (EMG) signals output from EMG sensors 31-1 to 31-8 are used to obtain EMG data for various finger shapes, such as: hand and finger shapes in an open hand state, hand and finger shapes in a hand-gripping state, and hand and finger shapes in a hand-pinching state. Then, the hand and finger shape recognition unit 51 can recognize the user's hand and finger shapes using an EMG data as input and a recognition model generated through machine learning (e.g., k-nearest neighbor method, support vector machine, neural network, etc.). In other words, the hand and finger shape recognition unit 51 can input EMG data supplied from the communication unit 41 into the recognition model and output a recognition result indicating the user's hand and finger shapes identified from the EMG data.

[0052] The force state recognition unit 52 identifies the force state of the user's fingers based on the myoelectric potential data supplied from the communication unit 41, and supplies the recognition result indicating the force state of the user's fingers to the initial motion detection unit 53 and the operation execution unit 55.

[0053] For example, such as Figure 4 As shown, the force state recognition unit 52 can identify the force state of the user's finger based on the threshold of the electromyographic data. That is, it identifies the force state of the user's finger based on whether the peak value of the electromyographic data obtained from the electromyographic signals output from the electromyographic sensors 31-1 to 31-8 is equal to or greater than a preset threshold shown by alternating long and short dashed lines.

[0054] Specifically, in Figure 4 In the example shown, when all peak values ​​of the myoelectric data corresponding to myoelectric sensors 31-1 to 31-8 are equal to or less than the threshold, the force state is identified as the hand being extended on the table in a static state. Among the myoelectric data corresponding to myoelectric sensors 31-1 to 31-8, when the peak values ​​of the myoelectric data corresponding to myoelectric sensors 31-1, 31-3, and 31-6 to 31-8 are equal to or greater than the threshold, the force state is identified as applying force with the index finger and the hand being extended on the table. Among the myoelectric data corresponding to myoelectric sensors 31-1 to 31-8, when the peak values ​​of the myoelectric data corresponding to myoelectric sensors 31-1 to 31-7 are equal to or greater than the threshold, the force state is identified as applying force with the thumb and the hand being extended on the table.

[0055] Alternative locations, such as Figure 5 As shown, the force state recognition unit 52 can identify the force state of a user's fingers by using a recognition model that takes both electromyographic (EMG) data of the unforced state and EMG data of the applied force states of all five fingers as input. This recognition model is generated through machine learning (e.g., k-nearest neighbor method, support vector machine, neural network, etc.). In other words, the force state recognition unit 52 can input EMG data supplied from the communication unit 41 into the recognition model and output a recognition result indicating the force state of the user's fingers identified from the EMG data. Note that when the force state recognition unit 52 uses the recognition model to identify the force state of the user's fingers, the force state of the user's fingers can be combined with the recognition model used by the hand and finger shape recognition unit 51 to form a single recognition model.

[0056] The initial action detection unit 53 detects that the user has performed an initial action based on the user's hand and finger shape identified by the hand and finger shape recognition unit 51 and the force state of the user's fingers identified by the force state recognition unit 52. For example, the user performs an initial action when he / she wants to start operating the information processing terminal 13 by applying force with his / her fingers.

[0057] Here, the initial action uses a combination of finger-applying gestures that would not normally be performed when the user grasps the real object 21. This action can be detected based on myoelectric potentials measured by the myoelectric potential measuring device 12. The action form of the initial action is pre-registered in the registration data of the storage unit 42, and the initial action can be registered as any action form. For example, different initial actions can be registered in the registration data depending on the real object 21 to be grasped. Note that any object that can be grasped in the real world can be used as the real object 21, and for example, the information processing terminal 13 can be used as the real object 21, or a part of the user's body can be used as the real object 21.

[0058] Figure 6 An example of the initial movement is shown, and the numbers indicated indicate the order in which the fingers applied force. For example, as... Figure 6 As shown in Figure A, the action of applying force sequentially from the index finger, middle finger, ring finger, little finger, and thumb in the shape of the hand and fingers grasping the stick-shaped real object 21A can be used as the initial action. Additionally, as... Figure 6 As shown in B, the initial action can be performed by using the hand and finger shape of the thumb, index finger and middle finger to pinch the real object 21B of the small cuboid, without applying force to the real object 21B, while applying force sequentially with the ring finger and little finger.

[0059] In addition, as an initial action, you can use the following forms: applying force with the index and little fingers at the same time, applying force three times in a short period of time with the thumb, applying force with the index and thumb for five seconds, applying force with the fingers at one-second intervals, or applying force in two stages, starting with a weak force and then applying a strong force.

[0060] For example, the primary purpose of the initial action is to trigger an operation input to the information processing terminal 13. This prevents operation input that contradicts the user's intention and clearly indicates the start of operation input to the information processing terminal 13. Furthermore, in the terminal operating system 11, after detecting the initial action, electromyographic (EMG) data for identifying the user's hand and finger shape and the force state of the fingers is acquired at a high sampling rate, thus suppressing power consumption. For example, in the terminal operating system 11, in order to acquire EMG data for detecting the initial action at a low sampling rate, it is preferable to use an action that involves applying force with the fingers for a prolonged period, which allows for acquisition even at a low sampling rate, as the form of the initial action.

[0061] The second purpose of the initial movement is to use it as a calibration (initialization process) to set a threshold for the operation execution unit 55 to detect the force state of the finger. For example, the maximum value of the myoelectric potential data when the finger applies force is stored in the initial movement, and any percentage of these maximum values ​​can be set as a threshold used by the operation execution unit 55.

[0062] For example, Figure 7 The left side shows an example of myoelectric data obtained from the myoelectric signals output from myoelectric sensors 31-1 to 31-8 when the user is in a static state during the initial movement, and this myoelectric data is recorded as myoelectric data indicating the shape of the hand and fingers during the initial movement. Figure 7 The right side shows an example of myoelectric potential (MP) data obtained from the MP signals output by MP sensors 31-1 to 31-8 when force is first applied with a finger in an initial movement from a resting state. In this way, the maximum value of the MP data when force is first applied with a finger is stored, and any percentage of the maximum value, as shown by alternating long and short dashed lines, can be set as a threshold for the operation execution unit 55 to detect the force state when force is applied with a finger.

[0063] Note that in the terminal operating system 11, the actual object 21 grasped or supported by the user can be identified based on the form of the initial action. Different initial actions can then be associated with different actual objects 21 grasped or supported in the same form. Furthermore, in the terminal operating system 11, the frame rate for measuring myoelectric potentials can be determined based on the result of the initial movement, and for example, a lower frame rate can be set when myoelectric potentials fluctuate significantly and are easily identifiable.

[0064] When the user's hand and finger shape recognition unit 51 identifies the user's hand and finger shape and registers it in the registration data stored in the storage unit 42, the operation mode selection unit 54 selects the operation mode associated with the hand and finger shape. Conversely, if the user's hand and finger shape recognition unit 51 does not identify the user's hand and finger shape in the registration data stored in the storage unit 42, the operation mode selection unit 54 selects the standard mode as the operation mode. Note that, in addition to the operation mode selection unit 54 selecting the operation mode, the user can directly operate the information processing terminal 13 to select the desired operation mode. Furthermore, the operation modes registered in the registration data associated with the user's hand and finger shape in the storage unit 42 can be arbitrarily set, and the operation content of each operation mode can also be arbitrarily set.

[0065] Note that in the terminal operating system 11, even if the user does not perform the initial action, when the user's hand and finger shape is estimated to match the hand and finger shape during the pre-registered initial action based on the myoelectric data, the operation mode selection unit 54 can select an operation mode corresponding to the hand and finger shape.

[0066] When the operation execution unit 55 detects that the force state of the user's finger, as identified by the force state recognition unit 52 after the initial action is detected, corresponds to the force state of the operation content in the operation mode selected by the operation mode selection unit 54, the operation execution unit 55 performs processing according to the operation content corresponding to the force state. For example, the operation execution unit 55 may detect the force state of the user's finger based on a binary value (0 or 1) indicating whether the force of each finger exceeds a threshold. Alternatively, the operation execution unit 55 may detect the force state of the user's finger based on the linear output (0 to 100%) of the force of each finger.

[0067] Reference Figure 8 and Figure 9 Describe the shape of the hand and fingers, the operation mode, and the operation content.

[0068] like Figure 8As shown, when the user's hand and finger shape indicate that their right hand is open and all fingertips are in contact with the object, the operation mode selection unit 54 selects the mouse operation mode as the operation mode. In mouse operation mode, the operation execution unit 55 executes the processing corresponding to the operation content corresponding to the left click based on the force state applied by the index finger, and executes the processing corresponding to the operation content corresponding to the double click based on the force state applied continuously by the index finger. Similarly, in mouse operation mode, the operation execution unit 55 executes the processing corresponding to the operation content corresponding to the right click based on the force state applied by the middle finger, executes the processing corresponding to the operation content corresponding to the back button click based on the force state applied by the thumb, and executes the processing corresponding to the operation content corresponding to the forward button click based on the force state applied by the ring finger.

[0069] When the user's hand and finger shape indicate that the left hand is open and the fingertips (excluding the little finger) are in contact with the object, the operation mode selection unit 54 selects the D-pad operation mode as the operation mode. In the D-pad operation mode, the operation execution unit 55 executes processing based on the force applied by the middle finger, corresponding to the upward direction, and based on the force applied by the ring finger, corresponding to the leftward direction. Similarly, in the D-pad operation mode, the operation execution unit 55 executes processing based on the force applied by the index finger, corresponding to the rightward direction, and based on the force applied by the thumb, corresponding to the downward direction.

[0070] When the user grips the handlebars according to the shape of their hand and fingers, the operation mode selection unit 54 selects the map operation mode as the operation mode. In map operation mode, the operation execution unit 55 performs processing based on zooming in on the map by changing the force applied with the index finger to the middle finger, and performs processing based on zooming out on the map by changing the force applied with the middle finger to the index finger. Similarly, in map operation mode, the operation execution unit 55 performs processing based on playing the next guide by continuously applying force with the thumb.

[0071] When the user's hand and finger shape indicate that the earphone is held between the index finger and thumb, the operation mode selection unit 54 selects the volume operation mode as the operation mode. In the volume operation mode, the operation execution unit 55 performs processing for increasing the volume based on the force applied with the index finger, and performs processing for decreasing the volume based on the force applied with the thumb. Similarly, in the volume operation mode, the operation execution unit 55 performs processing for stopping or playing based on the force applied with all fingers.

[0072] like Figure 9 As shown in Figure A, if the user's hand and finger shapes are not registered in the registration data stored in storage unit 42, operation mode selection unit 54 selects standard mode as the operation mode. In standard mode, operation execution unit 55 performs processing based on the force state applied by the index finger, processing based on the force state applied by the middle finger, and processing based on the force state applied by the thumb.

[0073] like Figure 9 As shown in B, when the user's hand and finger shape is the OK gesture (a gesture forming a ring with the thumb and index finger), the operation mode selection unit 54 selects the yes-no mode as the operation mode. In the yes-no mode, the operation execution unit 55 performs processing according to the force state of the index finger and thumb, corresponding to the yes operation, and performs processing according to the force state of the middle finger and thumb, corresponding to the no operation.

[0074] Note, refer to Figure 8 and Figure 9 The described hand and finger shapes, operation patterns, and operation content are examples, and other finger shapes, operation patterns, and operation content can be used.

[0075] As described above, the terminal operating system 11 can enable the user to perform an initial action using the force of the user's five fingers while the user is grasping or supporting any real object 21, and can be set to a state where operation input to the information processing terminal 13 can begin. Then, the terminal operating system 11 can perform processing based on the user's hand and finger shape and the force state of the fingers, for example, based on different processing for each of the five fingers that exert force on the user.

[0076] Furthermore, the terminal operating system 11 can use any real object 21 that can be grasped in the real world to input operations onto the information processing terminal 13. For example, in the terminal operating system 11, even when using different real objects 21, if the combination of hand and finger shapes and the fingers used to apply force are the same, processing can be performed according to the same operation content. That is to say, the terminal operating system 11 is not limited to using known real objects 21 (e.g., a pen-shaped elongated object) to operate the information processing terminal 13. Therefore, the terminal operating system 11 can achieve more flexible operability by using any real object 21.

[0077] Examples of processing for electromyography measurement and operation execution

[0078] Reference Figure 10 and Figure 11The flowchart shown describes the myoelectric potential measurement processing and operation execution processing performed in the terminal operating system 11.

[0079] Reference Figure 10 The flowchart shown describes the myoelectric potential measurement process performed by the myoelectric potential measurement device 12.

[0080] In step S11, the communication unit 34 determines whether it has received a signal indicating the start of communication sent from the communication unit 41 of the information processing terminal 13, and waits until it is determined that the signal indicating the start of communication has been received. Then, if the communication unit 34 determines that the signal indicating the start of communication has been received, the process proceeds to step S12.

[0081] In step S12, the signal acquisition unit 32 acquires the myoelectric potential signals output from the myoelectric potential sensors 31-1 to 31-8 and supplies the myoelectric potential signals to the signal processing unit 33.

[0082] In step S13, the signal processing unit 33 acquires myoelectric potential data as a result of performing signal processing (e.g., noise removal processing and Fourier transform processing) on ​​the myoelectric potential signal supplied in step S12, and supplies the myoelectric potential data to the communication unit 34.

[0083] In step S14, the communication unit 34 sends the myoelectric potential data supplied in step S13 to the information processing terminal 13.

[0084] In step S15, the communication unit 34 determines whether it has received a signal indicating the end of communication sent from the communication unit 41 of the information processing terminal 13. If the communication unit 34 determines that it has not received a signal indicating the end of communication, the process returns to step S12 and repeats the same process. On the other hand, if the communication unit 34 determines that it has received a signal indicating the end of communication, the process proceeds to step S16.

[0085] In step S16, after performing various processing operations to terminate the operation of the electromyography (EMG) measuring device 12, the EMG measurement process is terminated.

[0086] Reference Figure 11 The flowchart shown describes the operation execution process performed by the information processing terminal 13.

[0087] In step S21, the communication unit 41 sends a signal indicating the start of communication to the communication unit 34 of the electromyography measuring device 12.

[0088] In step S22, the communication unit 41 receives the data from the communication unit. Figure 10In step S14, the myoelectric potential data is sent from the communication unit 34 and supplied to the hand and finger shape recognition unit 51 and the force state recognition unit 52.

[0089] In step S23, the hand and finger shape recognition unit 51 recognizes the user's hand and finger shape based on the myoelectric data supplied in step S22, and supplies the recognition result indicating the user's hand and finger shape to the initial motion detection unit 53 and the operation mode selection unit 54.

[0090] In step S24, the force state recognition unit 52 identifies the force state of the user's fingers based on the myoelectric data supplied in step S22, and supplies the recognition result indicating the force state of the user's fingers to the initial motion detection unit 53 and the operation execution unit 55.

[0091] In step S25, the initial motion detection unit 53 determines whether to detect the user's initial motion based on the shape of the user's hand and fingers identified in step S23 and the force state of the user's fingers identified in step S24.

[0092] In step S25, if the initial action detection unit 53 determines that the user's initial action has been detected, the process proceeds to step S26.

[0093] In step S26, the operation mode selection unit 54 determines whether the user's hand and finger shapes identified in step S23 have been registered in the registration data stored in the storage unit 42.

[0094] In step S26, if the operation mode selection unit 54 determines that the user's hand and finger shapes are registered in the registration data, the process proceeds to step S27. In step S27, the operation mode selection unit 54 selects an operation mode associated with the user's hand and finger shapes identified in step S23 from the registration data, and the process proceeds to step S30.

[0095] On the other hand, in step S26, if the operation mode selection unit 54 determines that the user's hand and finger shapes are not registered (not registered) in the registration data, the process proceeds to step S28. In step S28, the operation mode selection unit 54 selects the standard mode as the operation mode, and the process proceeds to step S30.

[0096] On the other hand, in step S25, if the initial action detection unit 53 determines that no user initial action has been detected, the process proceeds to step S29.

[0097] In step S29, the operation mode selection unit 54 determines whether the user's hand and finger shape identified in step S23 matches the hand and finger shape at the time of the initial action. Then, if the operation mode selection unit 54 determines that the user's hand and finger shape does not match the hand and finger shape at the time of the initial action, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if the operation mode selection unit 54 determines that the user's hand and finger shape matches the hand and finger shape at the time of the initial action, the operation mode corresponding to the hand and finger shape is selected, and the process proceeds to step S30.

[0098] In step S30, after detecting the initial action, for the force state of the user's finger identified by the force state recognition unit 52, the operation execution unit 55 determines whether a force state corresponding to the force state of the operation content in the selected operation mode is detected.

[0099] In step S30, if the operation execution unit 55 determines that no force state corresponding to the force state of the operation content in the selected operation mode is detected, the process returns to step S22, and the same process is repeated thereafter. On the other hand, if in step S30, the operation execution unit 55 determines that a force state corresponding to the force state of the operation content in the selected operation mode is detected, the process proceeds to step S31.

[0100] In step S31, the operation execution unit 55 performs processing according to the operation content corresponding to the force state of the user's finger detected in step S30.

[0101] In step S32, the arithmetic unit 43 determines whether to end the operation execution process.

[0102] In step S32, if the arithmetic unit 43 determines that the operation execution process has not yet ended, the process returns to step S22, and the above process is repeated. On the other hand, if the arithmetic unit 43 determines that the operation execution process has ended, the process proceeds to step S33.

[0103] In step S33, the communication unit 41 sends a signal indicating the end of communication to the communication unit 34 of the electromyography measuring device 12, and then the operation execution process ends.

[0104] As described above, the terminal operating system 11 can achieve greater flexibility in operation by using any real object 21.

[0105] Examples of using and applying terminal operating systems

[0106] Reference Figures 12 to 15 This section describes usage examples and application examples of the terminal operating system 11.

[0107] Figure 12 This is a diagram illustrating an example of using a smartwatch as the terminal operating system 11.

[0108] For example, in the case of using a smartwatch as the terminal operating system 11, in Figure 12 In the illustrated usage example, the electromyography (EMG) measuring device 12 is included in the strap of a smartwatch, and the aforementioned operations are performed by the information processing terminal 13 located in the main body of the smartwatch. Figure 12 In the usage example shown, the handlebars of a bicycle are used as the real object 21C.

[0109] like Figure 12 As shown in A, when the user's hand and finger shape indicate gripping the handlebars, the map operation mode is selected as the operation mode.

[0110] like Figure 12 As shown in B, when a user launches a map application on a smartwatch that serves as the terminal operating system 11 in order to cycle to a destination, a map is displayed in the display area of ​​the terminal operating system 11, and navigation to the destination begins. Then, the user begins to input operations into the terminal operating system 11, for example, by applying force simultaneously with the index and little fingers, and then simultaneously with the middle and ring fingers, thereby starting to cycle.

[0111] For example, such as Figure 12 As shown in C, when a user wants to know when the next right turn will occur, the user applies continuous force with their thumb, and then, based on the operation corresponding to the force state, the navigation "Turn right 450m ahead" is displayed in the display area of ​​the terminal operating system 11.

[0112] like Figure 12 As shown in D, when a user wants to confirm a destination on the map but cannot display the destination at the current map scale, if the user applies pressure with their index finger and then with their middle finger, the map displayed on the terminal operating system 11 is magnified according to the operation corresponding to the pressure applied. Note that the map can be magnified so that the magnification changes linearly according to the intensity of the pressure applied by the fingers at this time.

[0113] Figure 13 This is a diagram illustrating a user's use of AR glasses as an information processing terminal 13 and a spatula as a real object 21D for cooking.

[0114] For example, a user wears AR glasses (which function as information processing terminal 13) on their head to play a moving recipe image, grips a spatula with their right hand (equipped with an electromyography device 12), and holds a frying pan with their left hand to cook. At this point, there might be a scenario where the user wants to return the moving recipe image to the displayed ingredient list to confirm the ingredients to be added next, but they cannot remove their hand because the food might burn. In such a case, the user performs the initial action by sequentially applying force with their right hand using their index, middle, and thumb fingers, while gripping the spatula.

[0115] At this time, if the hand shape for maintaining the spatula is not registered in the registration data of storage unit 42, select ( Figure 13 A) Standard mode is used as the action mode. Then, as... Figure 13 As shown in B, on the AR glasses that serve as the information processing terminal 13, operation guides, as shown by the dotted lines, are displayed on each finger in a manner that reflects the operation content of the standard mode. That is, an operation guide indicating the operation content "forward" is displayed on the index finger, an operation guide indicating the operation content "backward" is displayed on the middle finger, and an operation guide indicating the operation content "replay" is displayed on the thumb.

[0116] Therefore, when the user applies force with their right middle finger while gripping the spatula, the playback position of the recipe animation is returned to its previous position for 30 seconds, according to the corresponding action. This process is repeated with the middle finger until the playback position returns to the desired scene. Then, when the user applies force with their thumb when the playback position returns to the desired scene, the recipe animation is played according to the corresponding action.

[0117] Note that in the terminal operating system 11, when a user successfully performs an operation by applying force with their finger, the AR glasses, acting as the information processing terminal 13, can notify the user via video, sound, or vibration. For example, Figure 13 As shown in C, if a user successfully performs the "backward" action by applying pressure with their middle finger, a notification can be given by reducing the display of the operation guide. Alternatively, for example, a notification could be given for each finger in a different color, or a notification could be given for each finger in a different tone.

[0118] Furthermore, as operation on the terminal operating system 11 progresses over time, assuming the user is fatigued and the intensity of the force applied by the user with his / her fingers decreases, a threshold for the intensity of the force required for operation can be reduced from the start of operation over time.

[0119] For example, such as Figure 14As shown in A, since no time has elapsed since the start of the operation, the operation guide is displayed on the user's fingers, which are gripping the stick-shaped real object 21E, in a superimposed manner at its usual size. Then, as... Figure 14 As shown in B, when a certain amount of time has elapsed since the start of the operation, the operation guide displayed on the user's fingers gripping the stick-shaped real object 21E in an overlay manner is smaller than the usual size to indicate that the threshold of force required for the operation has decreased. In this case, conversely, the operation guide can be displayed in a larger size than usual to encourage the user to operate with greater force.

[0120] An application example will be described where a camera device is installed on an information processing terminal 13 and can be used to detect a real object 21.

[0121] For example, when a known real object 21 is identified by the camera device, the information processing terminal 13 can select an action mode corresponding to the known real object 21 without performing an initial action. On the other hand, when an unknown real object 21 is identified by the camera device, the information processing terminal 13 can select a standard mode without performing an initial action. Alternatively, if AR overlay display is possible, the information processing terminal 13 can display a hand icon overlaid on the real object 21 identified by the camera device, and display the operation content that can be performed using the real object 21. Furthermore, if the user's hand can be detected by the camera device, the information processing terminal 13 can identify the hand and finger shapes not only using electromyographic data but also using images of the hand captured by the camera device.

[0122] The information processing terminal 13 can use a camera device to identify the relative position of the user's finger applying force and adjust the processing accordingly. For example, as Figure 15 As shown, when performing operations on the keyboard, even when applying the same force with the index finger, the processing to be performed can be changed according to the position, so that when the index finger applies force at a certain position, the operation of inputting "J" is performed, and when the index finger applies force at a position 2cm to the right from that position, the operation of inputting "K" is performed.

[0123] Furthermore, as an application example, in the terminal operating system 11, within a certain period of time after a certain action occurs on the information processing terminal 13 side, the terminal operating system 11 can assume that an operation input needs to be performed, and even without performing the initial action, select a specific operating mode based on electromyographic data and accept the operation from the information processing terminal 13. Then, when receiving data to start operation in the specific operating mode, the sampling frequency used to measure the electromyographic data can be increased.

[0124] In the terminal operating system 11, when the information processing terminal 13 is a smartphone, when a voice message (e.g., an incoming call) is received, operations on the information processing terminal 13 based on electromyographic data can be received for a certain period thereafter. At this time, for example, it is assumed that a response or rejection operation is performed based on the electromyographic data, and therefore a yes-no mode can be selected (see [reference]). Figure 9 B).

[0125] Furthermore, in the terminal operating system 11, preparations are made for unintentional user operations, and the operation to cancel the previous operation input (cancellation operation) can be set to be available in all operating modes. For example, as a cancellation operation, consider the force state of using all fingers to open, the force state of applying force with all fingers, etc. In addition, in the terminal operating system 11, when a certain number or more cancellation operations are performed, the operation input to be cancelled can be regarded as an error detection, and the execution of processing based on the operation input can be stopped. Moreover, by learning the results of cancellation operations, the operating mode can be corrected.

[0126] Furthermore, the terminal operating system 11 can be used for motion analysis. For example, when gripping a golf club (real object 21), the terminal operating system 11 can store the way force is applied with each finger instead of explicit operation input, analyze the way force is applied as a golf motion, and provide feedback such as "Please relax your thumb." Similarly, the terminal operating system 11 can measure the user's level of concentration on driving, etc., based on how the user grips the steering wheel (real object 21) while driving.

[0127] The terminal operating system 11 can identify an individual based on the gripping posture of the real object 21, the way force is applied when gripping the real object 21, and so on. For example, processing can be performed by referring to the registration data of each individual.

[0128] Note that as long as the terminal operating system 11 can recognize the shape of the user's hand and fingers and the force state of the fingers, in addition to myoelectric data, the terminal operating system 11 can also use measurement data obtained by using sensors to detect changes in myoelectric sounds, ultrasound, electromagnetic waves, capacitance, etc.

[0129] Furthermore, the terminal operating system 11 can be configured such that some blocks of the storage unit 42 and the processing unit 43 are located in a server at a remote location, and the server and the information processing terminal 13 are always connected via a network. In this configuration, the information processing terminal 13 needs to have the function of synchronizing with the remote server via wired communication, wireless communication, etc. In addition, in this configuration, since the server at the remote location stores the action form of the initial action, the correspondence between the user's hand and finger shapes and the operation mode, the operation content of each operation mode, etc., the same operation input can be performed using the same real object 21 even when using different information processing terminals 13.

[0130] Computer configuration examples

[0131] Next, the aforementioned series of processing operations (information processing methods) can be performed by hardware or by software. In the case where the series of processing operations are performed by software, the program constituting the software is installed in a general-purpose computer or similar device.

[0132] Figure 16 This is a block diagram illustrating a configuration example of a computer in which a program for performing the above-described series of processing operations is installed.

[0133] In the computer, the central processing unit (CPU) 101, read-only memory (ROM) 102, random access memory (RAM) 103, and electrically erasable programmable read-only memory (EEPROM) 104 are connected to each other via bus 105. Bus 105 is also connected to input / output interface 106, and input / output interface 106 is connected to the outside.

[0134] In the computer configured as described above, the CPU 101 performs the aforementioned series of processing operations, for example, by loading a program stored in ROM 102 and EEPROM 104 into RAM 103 via bus 105 and executing the program. The program executed by the computer (CPU 101) can be pre-written into ROM 102, installed in EEPROM 104, or updated externally via input / output interface 106.

[0135] In this specification, the processes performed by the computer according to the program do not need to be executed sequentially in the order described in the flowchart. That is, the processes performed by the computer according to the program include processes executed in parallel or processes executed in a separate manner (e.g., parallel processing or processing performed by an object).

[0136] A program can be processed by a single computer (processor) or by multiple computers in a distributed manner. The program can be transmitted to a remote computer and executed there.

[0137] Furthermore, in this specification, "system" refers to a collection of multiple constituent elements (devices, modules (components), etc.), and it is not important whether all constituent elements are housed in the same housing. Therefore, "system" can refer to both multiple devices housed in a single housing and connected via a network, and multiple modules housed in a single housing for one device.

[0138] For example, a configuration described as a single device (or processing unit) can be partitioned and configured into multiple devices (or processing units). Conversely, a configuration described above as multiple devices (or processing units) can be integrated into a single device (or processing unit). It goes without saying that configurations other than those described above can be added to the configuration of each device (or each processing unit). When the configuration and operation of the entire system are substantially the same, the configuration of one device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0139] Additionally, for example, this technology can have a cloud computing configuration, in which one function is distributed and processed collaboratively by multiple devices via a network.

[0140] For example, the above procedure can be executed by any device. In this case, the device needs to have the necessary functions (function blocks, etc.) to obtain the necessary information.

[0141] For example, in addition to being executed by a single device, each step described in the flowchart above can be distributed and executed by multiple devices. Furthermore, when a single step includes multiple processing operations, the multiple processing operations included in that single step can be executed by a single device, or they can be distributed and executed by multiple devices. In other words, multiple processing operations included in a step can be executed as processing for multiple steps. Conversely, processing described as multiple steps can be executed together as a single step.

[0142] Note that a program executed by a computer can be configured such that the steps describing the program are executed sequentially in the order described in this specification, or in parallel, or separately at necessary timed intervals such as when called. That is, each step can be executed in a different order than described above, provided there is no contradiction. Furthermore, the steps describing the program can be executed in parallel with the processes of other programs, or in combination with the processes of other programs.

[0143] Note that each of the various techniques described in this specification can be implemented independently in a single component, provided that there is no contradiction. Of course, the various techniques described in this specification can be implemented in any combination of the various techniques. For example, part or all of the techniques described in any embodiment can be implemented in combination with part or all of the invention described in another embodiment. Furthermore, part or all of any of the techniques described above can be implemented in combination with another technique not described above.

[0144] Configuration combination examples

[0145] Note that this technology can also be configured as follows. (1)

[0147] An information processing apparatus, comprising:

[0148] Acquisition unit, configured to acquire measurement data corresponding to the movement of a user's hand and fingers in a state of grasping or supporting any real object; and

[0149] The processing unit is configured to estimate the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and to perform processing based on the user's operation. (2)

[0151] According to the information processing device of (1), the arithmetic unit includes

[0152] The computing unit includes

[0153] A hand and finger shape recognition unit, which identifies hand and finger shapes representing the user's hand and fingers based on the measurement data, and

[0154] A force state recognition unit, which identifies the finger applying force among the user's five fingers and the force state representing the intensity of the force applied by that finger based on the measurement data. (3)

[0156] According to the information processing device of (2), the computing unit further includes an initial action detection unit, which detects that the user has performed an initial action based on the user's hand and finger shape recognized by the hand and finger shape recognition unit and the force state of the user's fingers recognized by the force state recognition unit. (4)

[0158] According to the information processing device of (3), wherein the initial action uses the action form of a gesture, in which a user grasps or supports any real object by applying force with a predetermined combination of fingers. (5)

[0160] According to the information processing device of (3), the computing unit further includes an operation mode selection unit, which selects an operation mode associated with the user's hand and finger shape recognized by the hand and finger shape recognition unit when the initial action detection unit detects that the user has performed an initial action. (6)

[0162] According to the information processing device of (5), the arithmetic unit further includes an operation execution unit, which performs processing based on the operation content corresponding to the force state when it detects that the force state of the user's finger identified by the force state recognition unit is a force state corresponding to the force state of the operation content in the operation mode selected by the operation mode selection unit. (7)

[0164] According to the information processing device of (6), when the hand and finger shape recognition unit recognizes that the user's hand and finger shape is gripping the handle, the operation mode selection unit selects the map operation mode as the operation mode. (8)

[0166] According to the information processing apparatus described in (7), wherein, when the map operation mode is selected by the operation mode selection unit, the operation execution unit

[0167] Based on the force state changing from applying force with the index finger to applying force with the middle finger, the processing corresponding to the "map zoom-in" operation is performed.

[0168] Based on the force state changing from applying force with the middle finger to applying force with the index finger, the processing corresponding to the "map zoom-out" operation is performed, and...

[0169] Based on the force applied continuously with the thumb, the processing of the operation content corresponding to "play the next guide" is performed. (9)

[0171] According to the information processing device of (6), in the case that the hand and finger shape of the user identified by the hand and finger shape recognition unit is not registered, the operation mode selection unit selects the standard mode as the operation mode. (10)

[0173] According to the information processing device of (9), when the standard mode is selected by the operation mode selection unit, the operation execution unit

[0174] Based on the force applied by the index finger, the corresponding operation is executed.

[0175] Based on the force applied by the middle finger, the corresponding operation for "backward" is executed.

[0176] Based on the force applied by the thumb, the processing of the operation content corresponding to "OK" is performed. (11)

[0178] According to any one of (5) to (10) of the information processing apparatus, wherein, even if the user does not perform the initial action, if the hand and finger shape of the user identified by the hand and finger shape recognition unit matches the hand and finger shape at the time of the initial action, the operation mode selection unit selects the operation mode corresponding to the hand and finger shape. (12)

[0180] The information processing apparatus according to any one of (1) to (11), wherein the measurement data is myoelectric potential data obtained as a result of measuring the myoelectric potential of the user's arm. (13)

[0182] According to the information processing device of (12), wherein,

[0183] The information processing device is a smartwatch that includes a myocardial potential measurement device that outputs the myocardial potential data in the watchband, and

[0184] At least the computing unit is located in the main body of the smartwatch. (14)

[0186] An information processing method, comprising:

[0187] Measurement data corresponding to the movement of a user's hand and fingers while grasping or supporting any real object is acquired through an information processing device; and

[0188] The information processing device estimates the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and performs processing according to the user's operation. (15)

[0190] A program for causing a computer in an information processing device to perform information processing, comprising:

[0191] Acquire measurement data corresponding to the movement of the user's hand and fingers while grasping or supporting any real object; and

[0192] Based on the measurement data, the user's hand and finger shape and hand force state are estimated, and processing is performed according to the user's operation.

[0193] This embodiment is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of this disclosure. Furthermore, the effects described in this specification are merely illustrative and not restrictive, and other effects may be obtained.

[0194] Reference tag list

[0195] 11 Terminal operating system, 12 Myoelectric potential measurement device, 13 Information processing terminal, 21 Real object, 31 Myoelectric potential sensor, 32 Signal acquisition unit, 33 Signal processing unit, 34 Communication unit, 41 Communication unit, 42 Storage unit, 43 Computation unit, 51 Finger shape recognition unit, 52 Force state recognition unit, 53 Initial movement detection unit, 54 Operation mode selection unit, 55 Operation execution unit.

Claims

1. An information processing apparatus, comprising: The acquisition unit is configured to acquire measurement data corresponding to the movement of a user's hand and fingers in a state of grasping or supporting any real object; as well as The processing unit is configured to estimate the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and to perform processing based on the user's operation.

2. The information processing apparatus according to claim 1, wherein, The computing unit includes A hand and finger shape recognition unit, which identifies hand and finger shapes representing the user's hand and fingers based on the measurement data, and A force state recognition unit, which identifies the finger applying force among the user's five fingers and the force state representing the intensity of the force applied by that finger based on the measurement data.

3. The information processing apparatus according to claim 2, wherein, The computing unit further includes an initial action detection unit, which detects that the user has performed an initial action based on the user's hand and finger shape recognized by the hand and finger shape recognition unit and the force state of the user's fingers recognized by the force state recognition unit.

4. The information processing apparatus according to claim 3, wherein, The initial action uses a gesture, in which a user grasps or supports any real object by applying force with a predetermined combination of fingers.

5. The information processing apparatus according to claim 3, wherein, The computing unit further includes an operation mode selection unit, which selects an operation mode associated with the user's hand and finger shape identified by the hand and finger shape recognition unit when the initial action detection unit detects that the user has performed an initial action.

6. The information processing apparatus according to claim 5, wherein, The computing unit further includes an operation execution unit, which performs processing based on the operation content corresponding to the force state when it detects that the force state of the user's finger identified by the force state recognition unit is a force state corresponding to the force state of the operation content in the operation mode selected by the operation mode selection unit.

7. The information processing apparatus according to claim 6, wherein, When the hand and finger shape recognition unit recognizes that the user's hand and finger shape is gripping the handle, the operation mode selection unit selects the map operation mode as the operation mode.

8. The information processing apparatus according to claim 7, wherein, When the map operation mode is selected by the operation mode selection unit, the operation execution unit Based on the force state of changing from applying force with the index finger to applying force with the middle finger, the processing corresponding to the "map zoom-in" operation is performed. Based on the force state changing from applying force with the middle finger to applying force with the index finger, the processing corresponding to the "map zoom-out" operation is performed, and... Based on the force applied continuously with the thumb, the processing of the operation content corresponding to "play the next guide" is performed.

9. The information processing apparatus according to claim 6, wherein, If the user's hand and finger shape, as identified by the hand and finger shape recognition unit, is not registered, the operation mode selection unit selects the standard mode as the operation mode.

10. The information processing apparatus according to claim 9, wherein, When the standard mode is selected by the operation mode selection unit, the operation execution unit Based on the force applied by the index finger, the corresponding operation is executed. Based on the force applied by the middle finger, the corresponding operation for "backward" is executed. Based on the force applied by the thumb, the processing is performed according to the operation content corresponding to "OK".

11. The information processing apparatus according to claim 5, wherein, Even if the user does not perform the initial action, if the hand and finger shape of the user identified by the hand and finger shape recognition unit matches the hand and finger shape at the time of the initial action, the operation mode selection unit selects the operation mode corresponding to that hand and finger shape.

12. The information processing apparatus according to claim 1, wherein, The measurement data is myoelectric data obtained as a result of measuring the myoelectric potential of the user's arm.

13. The information processing apparatus according to claim 12, wherein, The information processing device is a smartwatch that includes a myocardial potential measurement device that outputs the myocardial potential data in the watchband, and At least the computing unit is located in the main body of the smartwatch.

14. An information processing method, comprising: The information processing device acquires measurement data corresponding to the movement of the user's hand and fingers in the state of grasping or supporting any real object; as well as The information processing device estimates the shape of the user's hand and fingers and the force state of the hand based on the measurement data, and performs processing according to the user's operation.

15. A program for causing a computer of an information processing apparatus to perform information processing, the information processing comprising: Acquire measurement data corresponding to the movement of the user's hand and fingers in a state of grasping or supporting any real object; as well as Based on the measurement data, the user's hand and finger shape and hand force state are estimated, and processing is performed according to the user's operation.

Images