Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) comprises: a detection unit (110) and an estimation unit (152). The detection unit (110) detects, in a non-contact manner, angles of at least a part of joints of fingers. The estimation unit (152) estimates postures of the fingers on the basis of the angles of the joints detected by the detection unit (110).

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is known a technique for recording and reproducing finger movements in order to convey the excellent dexterity of musical instrument players, traditional craftsmen, chefs, surgeons, etc. to others (such as students) and to support others in improving their skills. For example, there is known a technique for measuring finger movements by attaching small sensors equipped with a 3-axis acceleration sensor and a 3-axis gyro sensor to each joint of the fingers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-54483 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology does not necessarily allow the movement of fingers to be measured appropriately. For example, when performing dexterous finger movements such as playing a musical instrument or performing a surgical operation, a fine sense of the fingertips is required. However, for example, in the above-mentioned conventional technology, a sensor needs to be attached to the joints of the fingers in order to measure the angle of the joints of the fingers, and the attachment of the sensor may hinder the dexterous movement of the fingers. Therefore, the above-mentioned conventional technology does not necessarily allow the movement of the fingers to be measured appropriately.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program capable of appropriately measuring finger movements. [Means for solving the problem]

[0006] In order to solve the above problems, an information processing device of one embodiment according to the present disclosure includes a detection unit that detects the angles of at least some of the joints of a finger in a non-contact manner, and an estimation unit that estimates the posture of the fingers based on the angles of the joints detected by the detection unit. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating an example of information processing according to the first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of an information processing system according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a finger sensor according to the embodiment. [Diagram 5] 11 is a diagram for explaining a unit of a finger portion of the finger sensor according to the embodiment, showing the entire unit as viewed from the side. FIG. [Figure 6] 13 is a diagram for explaining a unit of a finger portion of the finger sensor according to the embodiment, and is a diagram showing an enlarged view of the fingertip side portion of the unit as viewed from the side. FIG. [Figure 7] 11 is a diagram showing an example of a method for calculating a joint angle of a finger according to the embodiment; FIG. [Figure 8] 11 is a diagram showing an example of a method for calculating the adduction and abduction angles of the fingers according to the embodiment. FIG. [Figure 9] FIG. 2 is a perspective view of the exoskeleton robot according to the embodiment. [Figure 10] FIG. 2 is a side view of the exoskeleton robot according to the embodiment. [Figure 11] 13 is a diagram for explaining an exoskeleton unit corresponding to one finger of the exoskeleton robot according to the embodiment. FIG. [Figure 12] 13 is a diagram for explaining the RCM mechanism provided in the exoskeleton unit corresponding to one finger of the exoskeleton robot according to the embodiment. FIG. [Figure 13] FIG. 2 is a top view of the exoskeleton robot according to the embodiment. [Figure 14] 11 is a flowchart showing an information processing procedure according to the embodiment. [Figure 15] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 16] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 17] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 18] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 19] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 20] 13 is a diagram showing an example of a finger sensor according to a modified example of the embodiment. FIG. [Figure 21] FIG. 13 is a perspective view of an exoskeleton robot according to a modified example of the same embodiment. [Figure 22] FIG. 13 is a side view of an exoskeleton robot according to a modified example of the same embodiment. [Diagram 23] FIG. 13 is a top view of an exoskeleton robot according to a modified example of the same embodiment. [Figure 24] FIG. 13 is a bottom view of an exoskeleton robot according to a modified example of the same embodiment. [Diagram 25] FIG. 11 is a diagram illustrating an example of information processing according to a second embodiment of the present disclosure. [Figure 26] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 27] FIG. 2 is a diagram for conceptually explaining the flow of information processing according to the embodiment. [Figure 28] 11 is a flowchart showing an information processing procedure according to the embodiment. [Figure 29] 11 is a flowchart showing an information processing procedure according to the embodiment. [Diagram 30] 10 is a flowchart showing an information processing procedure according to the embodiment. [Diagram 31] FIG. 13 is a diagram showing an example of a screen when a processing result according to the embodiment is displayed on a terminal device of a user [Diagram 32]FIG. 13 is a diagram showing an example of a screen when a processing result according to the embodiment is displayed on a terminal device of a user. [Diagram 33] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0009] The present disclosure will be described in the following order. 0. Introduction 1. First embodiment 1.1. Overview of the information processing system 1.2. Example of information processing device configuration 1.3. Example of operation of information processing system 1.4. Variations 1.4.1. Hand Sensor 1.4.2. Exoskeleton Robots 2. Second embodiment 2.1. Overview of the information processing system 2.2. Example of information processing device configuration 2.3. Example of operation of information processing system 3.Other 4.Effects 5. Hardware Configuration

[0010] [0. Introduction] Recording and playing back the exquisite hand movements of musicians, traditional craftsmen, chefs, and other skilled artisans is extremely important for transferring the skills of experts to others (such as apprentices). In addition, recording high-speed hand movements and showing them to users is extremely effective in transferring intuitive tacit knowledge when supporting the development of skills.

[0011] However, recording fast and precise finger movements requires high spatial and temporal resolution, but attaching a device directly to the fingers for precise measurement can impede dexterity.

[0012] Therefore, in this invention, we propose a system that uses a measurement device and an exoskeleton robot, which allows the device to be attached directly to the fingers while minimizing the interference with dexterous movements.

[0013] [1. First embodiment] [1.1. Overview of information processing system] From here, an overview of information processing according to the first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the first embodiment of the present disclosure. The information processing according to the first embodiment is performed by an information processing device 100 shown in Fig. 1.

[0014] Prior to the description of FIG. 1, a configuration of an information processing system according to a first embodiment of the present disclosure will be described with reference to FIG. 2. FIG. 2 is a diagram showing a configuration example of the information processing system according to the first embodiment of the present disclosure. As shown in FIG. 2, the information processing system 1 according to the first embodiment includes an information processing device 100, an application server 10, and a terminal device 20. The information processing device 100, the application server 10, and the terminal device 20 are connected to each other via a predetermined network N so as to be able to communicate with each other by wire or wirelessly. Note that the information processing system 1 shown in FIG. 2 may include a plurality of information processing devices 100, a plurality of application servers 10, and a plurality of terminal devices 20.

[0015] The application server 10 acquires information about the feature amounts of the fingers estimated by the information processing device 100 from the information processing device 100. When the application server 10 acquires the information about the feature amounts of the fingers, the application server 10 generates content for presenting the information about the feature amounts of the fingers to the user. Then, when the application server 10 generates the content, the application server 10 distributes the generated content to the terminal device 20.

[0016] The terminal device 20 is an information processing device used by a user. For example, the terminal device 20 may be a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or other device. In the example shown in FIG. 1, the terminal device 20 is a smartphone.

[0017] The terminal device 20 presents information on the posture and position of the fingers estimated by the information processing device 100 to the user. Specifically, the terminal device 20 transmits a content distribution request to the application server 10. Furthermore, the terminal device 20 receives the content from the application server 10. Upon receiving the content from the application server 10, the terminal device 20 outputs the content received from the application server 10.

[0018] The information processing device 100 is a server device that performs information processing according to the first embodiment. The information processing device 100 acquires sensor information on the intensity of light reflected by the finger from a photoreflector included in the finger sensor 110. The information processing device 100 also acquires sensor information on the absolute angle of the middle joint of the finger relative to the ground surface from a 9-axis attitude sensor of the middle joint included in the finger sensor 110. The information processing device 100 also acquires sensor information on the absolute angle of the back of the hand relative to the ground surface from a 9-axis attitude sensor of the back of the hand included in the finger sensor 110. Next, the information processing device 100 estimates the angle of the DIP joint, PIP joint, or MP joint of the finger as information on the feature amount of the finger based on the acquired sensor information. The information processing device 100 also presents the information on the estimated feature amount of the finger to the user. For example, the information processing device 100 transmits information on the estimated feature amounts of fingers to the application server 10, and presents the information on the estimated feature amounts of fingers to the user's terminal device 20 via the application server 10. In addition, the information processing device 100 uses the exoskeleton robot 120 to present the information on the estimated feature amounts of fingers to the user.

[0019] Hereinafter, an overview of information processing according to the first embodiment will be described with reference to FIG. 1. In FIG. 1, a sensor information processing unit 151 of the information processing device 100 acquires sensor information from a finger sensor 110. An estimation unit 152 of the information processing device 100 estimates information on the feature amounts of fingers based on the sensor information acquired by the sensor information processing unit 151. For example, the estimation unit 152 estimates information on the feature amounts of fingers such as the angles of the finger joints, the average angular velocity of the finger joints, the maximum angular velocity, and the range of motion of the finger joints. Next, when the estimation unit 152 estimates information on the feature amounts of fingers, the estimation unit 152 stores the estimated information on the feature amounts of fingers in the storage unit 130.

[0020] The application server 10 acquires information on the feature amount of the fingers from the storage unit 130. When the application server 10 acquires the information on the feature amount of the fingers, the application server 10 generates content for presenting the information on the feature amount of the fingers to the user based on the acquired information. For example, the application server 10 generates content corresponding to the screens shown in FIG. 31 and FIG. 32 described later. The application server 10 also generates content for displaying the finger movements at a slow or fast-forward speed. The application server 10 also generates content for outputting a sound that matches the finger movements when presenting the finger movements. For example, in the case of finger movements playing a piano, the application server 10 generates content for outputting a sound that corresponds to the keys that the fingers have struck. Next, when the application server 10 generates the content, the application server 10 distributes the generated content to the terminal device 20.

[0021] The terminal device 20 receives content from the application server 10. When the terminal device 20 receives the content, it outputs the received content. For example, the terminal device 20 displays the screens shown in Figs. 31 and 32 described below. The terminal device 20 also displays the finger movements at slow or fast forward speeds. When presenting the finger movements, the terminal device 20 outputs sounds that match the movements. For example, in the case of finger movements that involve playing the piano, the terminal device 20 outputs sounds that correspond to the keys that the fingers have pressed.

[0022] Meanwhile, the drive control unit 153 of the information processing device 100 acquires information on the feature amounts of the fingers from the storage unit 130. Then, upon acquiring the information on the feature amounts of the fingers, the drive control unit 153 generates a command for the exoskeleton robot 120 based on the acquired information. Then, upon generating the command for the exoskeleton robot 120, the drive control unit 153 controls the driving of the exoskeleton robot 120 based on the generated command.

[0023] [1.2. Example of configuration of information processing device] Next, the configuration of the information processing device according to the first embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the information processing device according to the first embodiment of the present disclosure. As shown in Fig. 3, the information processing device 100 according to the first embodiment includes a finger sensor 110, an exoskeleton robot 120, a storage unit 130, a communication unit 140, and a control unit 150.

[0024] (Hand sensor 110) Next, the configuration of the finger sensor according to the first embodiment of the present disclosure will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the finger sensor according to the first embodiment of the present disclosure. As shown in Fig. 4, the finger sensor 110 includes four first sensor units 110A for measuring the postures of the index finger, middle finger, ring finger, and little finger, respectively, a second sensor unit 110B for measuring the posture of the thumb, and a 9-axis posture sensor 110C for measuring a reference posture that is a reference for the joint angles of the fingers.

[0025] The first sensor units 110A are attached to the middle joints of the index finger, middle finger, ring finger, and little finger. In the example shown in FIG. 4, each first sensor unit 110A is attached to the upper surface (dorsiflexion side) of the middle joint of each finger by a ring-shaped band. As shown in FIG. 4, the width of each first sensor unit 110A is designed to be narrower than the width of each finger. In addition, the length of each first sensor unit 110A is designed to be about 10% longer on both the top and bottom than the length of the middle joint of each finger, and a photoreflector is provided in the part designed to be about 10% longer on both the top and bottom. In addition, a thin, square, small 9-axis attitude sensor is attached to the center of each first sensor unit 110A.

[0026] The second sensor unit 110B is attached to the upper surface (dorsiflexion side) of the proximal phalanx of the thumb. In the example shown in FIG. 4, a thumb measurement sensor is attached to the upper surface (dorsiflexion side) of the proximal phalanx of the thumb by a ring-shaped band. The shape of the second sensor unit 110B seen from above is rectangular. The width of the second sensor unit 110B is designed to be narrower than the width of the thumb. In addition, the length of the second sensor unit 110B is designed to be about 10% longer on the MP joint side than the length of the proximal phalanx of the thumb, and a photoreflector is provided in the part designed to be about 10% longer on the MP joint side. In addition, a thin, square, small 9-axis attitude sensor is attached to the center of the second sensor unit 110B.

[0027] The 9-axis attitude sensor 110C is attached to the back of the hand. In the example shown in Fig. 4, a thin, square, small 9-axis attitude sensor 110C is attached to the back of the hand. Specifically, the 9-axis attitude sensor 110C is attached between the MP joints of the index finger and middle finger on the base bone and the wrist, or in the area between them (the area AR1 shown in Fig. 4) so ​​as not to be affected by the curvature of the palm or the movement of the thumb.

[0028] In this way, the second attitude sensor (9-axis attitude sensor 110C) is provided on the back of the hand, on the base bones of the index finger and middle finger, in the area between the MP joint and the wrist. This allows the information processing device 100 to detect the inclination of the back of the hand with respect to the direction of gravity without being affected by the curvature of the palm or the movement of the thumb.

[0029] Next, a unit of a finger part of a finger sensor according to a first embodiment of the present disclosure will be described with reference to FIG. 5. FIG. 5 is a diagram for describing a unit of a finger part of a finger sensor according to a first embodiment of the present disclosure, and is a diagram showing the entire unit viewed from the side. In the example shown in FIG. 5, the first sensor unit 110A is provided with one 9-axis attitude sensor for measuring the absolute angle of the middle part with respect to the ground surface, and two photoreflectors for measuring the relative angle between the middle part and the distal part and the relative angle between the middle part and the proximal part, at the middle part of each finger. In the following, the relative angle between the middle part and the distal part may be referred to as the angle of the DIP joint. In addition, the relative angle between the middle part and the proximal part may be referred to as the angle of the PIP joint.

[0030] Next, the unit of the finger part of the finger sensor according to the first embodiment of the present disclosure will be further described with reference to FIG. 6. FIG. 6 is a diagram for explaining the unit of the finger part of the finger sensor according to the first embodiment of the present disclosure, and is a diagram showing an enlarged view of the distal phalangeal side of the unit from the lateral direction. In the example shown in FIG. 6, the first sensor unit 110A measures the distance between the sensor and the skin of the distal phalangeal by irradiating infrared light from a photoreflector attached on the extension of the upper surface of the middle phalangeal and detecting the intensity of the light reflected by the surface of the skin of the distal phalangeal and returning. Next, the first sensor unit 110A calculates the angle of the DIP joint based on the measured distance between the sensor and the skin of the distal phalangeal. In this way, the information processing device 100 can measure the angle of the PIP joint and the angle of the DIP joint in a non-contact manner by using a photoreflector.

[0031] In this way, the detection unit (finger sensor 110) detects the angles of at least some of the joints of the fingers in a non-contact manner. As a result, the information processing device 100 can detect the angles of the joints of the fingers without interfering with the fine sensation of the fingertips or the dexterous movements of the fingers, since the detection is non-contact. Therefore, the information processing device can appropriately measure the movements of the fingers.

[0032] The detection unit includes a non-contact sensor (photoreflector), detects the distance from the non-contact sensor to the finger, which changes depending on the angle of the joint, and detects the angle of the joint based on the detected distance. The non-contact sensor includes a light-emitting element that emits light and a light-receiving element that detects reflected light. The light-emitting element emits light to the joint of the finger. The light-receiving element detects the light reflected by the joint of the finger. The detection unit detects the distance to the finger based on the intensity of the reflected light detected by the light-receiving element. The non-contact sensor is not limited to an optical sensor (light sensor) such as a photoreflector, but may be any proximity sensor that can detect the distance from the sensor to the finger without contact, such as an ultrasonic sensor or a capacitance sensor. This makes it possible for the information processing device 100 to indirectly (i.e., without contact) detect the angle of the joint by detecting the distance from the non-contact sensor to the finger, instead of directly detecting the angle of the joint by attaching a sensor to the joint.

[0033] The non-contact sensor (photo reflector) is provided on at least one of the middle phalanges of the index finger, middle finger, ring finger, and little finger. The light emitting element irradiates light on at least one of the distal phalanges and the proximal phalanges of the fingers. The light receiving element detects light reflected by at least one of the distal phalanges and the proximal phalanges of the fingers. The detection unit calculates the relative angle of the distal phalange of the fingers to the middle phalange as the angle of the DIP joint of the fingers based on the distance between the distal phalange of the fingers and the non-contact sensor calculated based on the intensity of the reflected light. The estimation unit calculates the relative angle of the proximal phalange of the fingers to the middle phalange as the angle of the PIP joint of the fingers, and estimates the posture of the fingers based on the angle of the DIP joint and the angle of the PIP joint of the fingers. This allows the information processing device 100 to detect the angle of the DIP joint and the angle of the PIP joint of the fingers in a non-contact manner without attaching a sensor to the distal phalange and the proximal phalange of the fingers.

[0034] Furthermore, the information processing device 100 can measure the PIP joint angle and the DIP joint angle with relatively little constraint compared to conventional techniques, since it only requires attaching the small first sensor unit 110A and the small second sensor unit 110B to the middle joint and sticking the small 9-axis posture sensor 110C to the back of the hand.

[0035] Next, a method for calculating a joint angle of a finger according to the first embodiment of the present disclosure will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a method for calculating a joint angle of a finger according to the first embodiment of the present disclosure. Fig. 7 shows a model of one finger portion in which the palm, the proximal phalanges, the middle phalanges, and the distal phalanges of the fingers are regarded as links, and the MP joint between the palm and the proximal phalanges, the PIP joint between the proximal phalanges and the middle phalanges, and the DIP joint between the middle phalanges and the distal phalanges are regarded as connecting parts of the links.

[0036] Angle α0 shown in FIG. 7 indicates the absolute angle of the palm relative to the horizontal plane measured by the 9-axis posture sensor attached to the back of the hand. Angle α1 indicates the absolute angle of the middle joint relative to the horizontal plane measured by the 9-axis posture sensor attached to the middle joint. Angle θ0 indicates the angle of the MP joint, which is the relative angle between the palm and the proximal joint. Angle θ1 indicates the angle of the PIP joint, which is the relative angle between the proximal joint and the middle joint. Angle θ2 indicates the angle of the DIP joint, which is the relative angle between the middle joint and the distal joint. At this time, the relationship of "α0-θ0-θ1=α1" is established between each angle. Therefore, the estimation unit 152 calculates the angle θ0 of the MP joint to which the sensor is not attached based on the angle α0 and angle α1 measured by the 9-axis posture sensor, the angle θ1 calculated by the photoreflector, and the relational expression "α0-θ0-θ1=α1". This allows the information processing device 100 to calculate all angles of the joints of one finger using the 9-axis orientation sensor attached to the back of the hand and the first sensor unit 110A attached to the finger.

[0037] In this way, the information processing device 100 further includes a first posture sensor (a 9-axis posture sensor provided on the middle joint of the finger) for detecting the inclination of the middle joint of the finger with respect to the direction of gravity. The estimation unit calculates the angle of the MP joint of the finger based on the angle of the PIP joint of the finger and the inclination of the middle joint of the finger with respect to the direction of gravity detected by the first posture sensor. The information processing device 100 further includes a second posture sensor (a 9-axis posture sensor 110C) for detecting the inclination of the back of the hand with respect to the direction of gravity. The estimation unit calculates the relative angle of the proximal joint of the finger with respect to the back of the hand as the angle of the MP joint based on the inclination of the back of the hand with respect to the direction of gravity detected by the second posture sensor. This allows the information processing device to calculate the angle of the MP joint in a non-contact manner without attaching a sensor to the MP joint.

[0038] Next, a method for calculating the adduction and abduction angles of fingers according to the first embodiment of the present disclosure will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a method for calculating the adduction and abduction angles of fingers according to the first embodiment of the present disclosure. Fig. 8 is a model of an entire hand in which the joints of each finger are regarded as links and the joints of each finger are regarded as connecting parts of the links, similar to Fig. 7.

[0039] Angle β0 shown in FIG. 8 indicates the absolute angle of adduction or abduction of the palm with respect to the coordinate system of the earth's orientation with magnetic north as its axis measured by the 9-axis attitude sensor attached to the back of the hand. Angle φ1 indicates the relative angle of adduction or abduction of the fingers with respect to the reference coordinate system set on the back of the hand. Angle β1 indicates the absolute angle of adduction or abduction of the fingers with respect to the coordinate system of the earth's orientation with magnetic north as its axis measured by the 9-axis attitude sensor attached to the middle joint of the fingers. At this time, the relationship of "φ1=β1-β0" is established between the angles. Therefore, the estimation unit 152 calculates the relative angle φ1 of adduction or abduction of the fingers with respect to the reference coordinate system set on the back of the hand based on the angle β0 and angle β1 measured by the 9-axis attitude sensor and the relational expression of "φ1=β1-β0".

[0040] In this way, the first posture sensor (9-axis posture sensor provided on the middle joint of the finger) detects the inclination of the adduction or abduction of the fingers with respect to the direction of gravity. The second posture sensor (9-axis posture sensor attached to the back of the hand) detects the inclination of the adduction or abduction of the palm with respect to the direction of gravity. The estimation unit calculates information on the relative angle of the adduction or abduction of the fingers with respect to the back of the hand based on the inclination of the adduction or abduction of the fingers with respect to the direction of gravity detected by the first posture sensor and the inclination of the adduction or abduction of the palm with respect to the direction of gravity detected by the second posture sensor. This allows the information processing device 100 to estimate the angle of adduction or abduction of the fingers with relatively little constraint.

[0041] Next, a method for calculating the posture of the thumb will be described. The posture of the 9-axis posture sensor on the back of the hand, which is the posture reference, is expressed as a 3×3 rotation matrix M0. In addition, the posture of the 9-axis posture sensor attached to the thumb distal phalanx is expressed as a 3×3 rotation matrix M1. At this time, there exists a 3×3 rotation matrix R for which M0=RM1. The rotation matrix R is a rotation matrix that represents the three-dimensional relative angle of the 9-axis posture sensor attached to the thumb distal phalanx with respect to the posture of the 9-axis posture sensor on the back of the hand. In addition, the CM joint of the thumb has two degrees of freedom, and the MP joint has one degree of freedom. Since the directions of the rotation axes of each degree of freedom of the thumb do not coincide with each other, the rotation angle around each axis of the thumb can be calculated from the three-dimensional rotation matrix R. Specifically, the estimation unit 152 calculates the rotation matrix R based on the rotation matrices M0 and M1 measured by the 9-axis posture sensor and the relational expression of "M0=RM1".

[0042] The first attitude sensor is a nine-axis sensor that detects acceleration in three axial directions, angular velocity in three axial directions, and orientation in three axial directions. The second attitude sensor is a nine-axis sensor that detects acceleration in three axial directions, angular velocity in three axial directions, and orientation in three axial directions. This allows the information processing device 100 to appropriately detect the inclination of the middle joint of the finger relative to the direction of gravity and the inclination of the back of the hand relative to the direction of gravity.

[0043] (Exoskeleton Robot 120) The exoskeleton robot 120 reproduces the motion of the fingers detected by the finger sensor 110 under the control of the drive control unit 153. Specifically, the exoskeleton robot 120 includes a drive unit 121 and a sensor 122.

[0044] (Drive unit 121) The driving unit 121 expresses the movement of the fingers by the exoskeleton robot 120. For this purpose, the driving unit 121 includes the exoskeleton unit 120A and the motor 121A. For example, the driving unit 121 includes the exoskeleton unit 120A corresponding to one finger and the motor 121A for driving the exoskeleton unit 120A. In the example shown in FIG. 1 and FIG. 9 to FIG. 13, the driving unit 121 includes four exoskeleton units 120A corresponding to the index finger, the middle finger, the ring finger, and the little finger, respectively, and four motors 121A for driving the four exoskeleton units 120A, respectively. For example, the driving unit 121 adducts or abducts the exoskeleton unit 120A around the rotation axis AX1 according to the control of the drive control unit 153. Also, the driving unit 121 bends and straightens the exoskeleton unit 120A around the rotation axis of the RCM mechanism according to the control of the drive control unit 153.

[0045] (Sensor 122) The sensor 122 detects the posture and motion of the exoskeleton robot 120. Specifically, the sensor 122 is realized by a 9-axis posture sensor. For example, the sensor 122 detects the angle and angular velocity of the joints connecting the links of the exoskeleton unit 120A.

[0046] Next, an exoskeleton robot according to the first embodiment of the present disclosure will be described with reference to Fig. 9. Fig. 9 is a perspective view of the exoskeleton robot according to the first embodiment of the present disclosure. In the example shown in Fig. 9, the exoskeleton robot 120 includes four exoskeleton units 120A corresponding to the fingers for reproducing the movements of the index finger, middle finger, ring finger, and little finger, respectively, and four motors 121A for driving the four exoskeleton units 120A corresponding to the fingers, respectively.

[0047] As shown in FIG. 9, the exoskeleton robot 120 makes it possible to eliminate interference between the motor and the link mechanism by changing the length of the links of adjacent fingers by the size of the motor.

[0048] In the conventional exoskeleton robot, when the wearer tries to move the exoskeleton robot, the wearer may not be able to move it freely. The main reasons are that the frictional resistance of the drive transmission unit is large and the degree of freedom of adduction and abduction of the fingers is limited. The exoskeleton robot 120 according to the present invention solves these problems by using a powerful motor 121A (specifically, a torque motor) without a reduction gear and using a link mechanism in which two ball joints (a first ball joint J1 and a second ball joint J2) are arranged at both ends of the link of the exoskeleton unit 120A corresponding to one finger and connected to the RCM mechanism link unit. In addition, unlike the conventional exoskeleton robot, the exoskeleton robot 120 according to the present invention is less likely to hinder the movement of the wearer's fingers, so that the exoskeleton robot 120 can record the movement of the fingers by mounting a sensor on the exoskeleton robot 120.

[0049] Next, the exoskeleton robot according to the first embodiment of the disclosure will be further described with reference to FIG. 10. FIG. 10 is a side view of the exoskeleton robot according to the first embodiment of the present disclosure. As shown in FIG. 10, when the exoskeleton robot 120 is viewed from one side, an exoskeleton unit 120A corresponding to one finger is arranged on the rear side of the two motors 121A. The exoskeleton unit 120A corresponding to one finger includes an RCM (Remote Center of Motion) mechanism link part having a double parallelogram structure using parallel links. Here, RCM is an abbreviation for Remote Center of Motion. In general, an RCM mechanism is a mechanism that realizes pivot (fixed point) motion by arranging a rotation center at a position away from the rotation center of the motor. For example, an RCM mechanism having a double parallelogram structure has a degree of freedom to rotate around the rotation center of the RCM mechanism within a plane including the double parallelogram. The exoskeleton unit 120A also includes a first link L1, one end of which is fixed to a second vertex P2 diagonally located to the connection point PX of the two parallelogram structures of the RCM mechanism link section, a second link L2, one end of which is connected to the other end of the first link L1 by a first ball joint J1, and a third link L3, one end of which is connected to the other end of the second link L2 by a second ball joint J2, and the first link L1, the second link L2, and the third link L3 form a four-joint link structure. The exoskeleton unit 120A also includes a motor 121A, and the other end of the third link L3 is fixed to the rotation shaft of the motor 121A.

[0050] A wearer who wears the exoskeleton robot 120 wears the exoskeleton unit 120A on each of the index finger, middle finger, ring finger, and little finger such that the range from the MP joint to the fingertip of each finger is along the RCM mechanism link part. Specifically, the exoskeleton unit 120A is fixed to a first vertex P1 located diagonally opposite the connection point PX of the two parallelogram structures of the RCM mechanism link part, and includes a holding member that holds the fingers. The index finger, middle finger, ring finger, and little finger of the wearer of the exoskeleton robot 120 are held by the holding members of the exoskeleton unit 120A corresponding to each finger. The RCM mechanism link part extends and contracts in accordance with the movement of the wearer bending and straightening the fingers.

[0051] Next, an exoskeleton unit corresponding to one finger of the exoskeleton robot according to the first embodiment of the present disclosure will be described with reference to FIG. 11. FIG. 11 is a diagram for describing an exoskeleton unit corresponding to one finger of the exoskeleton robot according to the first embodiment of the present disclosure. FIG. 11 is a diagram showing an exoskeleton unit 120A corresponding to one finger from the opposite direction to FIG. 10. In the example shown in FIG. 11, the motor 121A is arranged on the back side of the exoskeleton unit 120A corresponding to one finger, opposite to FIG. 10. Here, the motor 121A is a direct drive or a motor with a small reduction ratio. This allows the wearer to freely move the finger of the exoskeleton robot 120 at their own will when it is not energized.

[0052] As shown in FIG. 11, the exoskeleton unit 120A includes a mechanism for adjusting the position of the MP joint. In this way, the exoskeleton unit 120A includes a mechanism for adjusting the position of the MP joint, so that the position of the MP joint of each finger can be adjusted with two degrees of freedom that are perpendicular to each other to match the shape of the wearer's hand. In addition, the exoskeleton unit 120A can fix the position by a screw after adjusting the position of the MP joint of each finger to match the shape of the wearer's hand. In this way, the exoskeleton unit 120A succeeded in fitting each unit to the width of the finger to fit the size of the hand. In addition, the exoskeleton robot 120 can adjust the rotation axis of the rotation mechanism that rotates the RCM mechanism in accordance with the rotation of the adduction or abduction of the fingers so that it is above the MP joint of each finger. As a result, the exoskeleton robot 120 makes it possible to move the original finger movements with almost no restriction.

[0053] The exoskeleton unit 120A is fixed to a first vertex P1 diagonally opposite to the connection point PX of the two parallelogram structures of the RCM mechanism link section, and includes a holding member for holding the fingers. The exoskeleton unit 120A may further include front and rear slider mechanisms as shown in the lower right of FIG. 11. For example, the exoskeleton unit 120A may include a slider mechanism for slidably holding a ring for fixing the fingers to the holding member. This allows the exoskeleton unit 120A to accommodate different lengths of the wearer's fingers.

[0054] 11, the exoskeleton unit 120A also includes a rotation axis AX1 for the degree of freedom of adduction or abduction of the fingers at a position just below the first ball joint J1. The exoskeleton unit 120A can measure the adduction or abduction of the fingers by embedding a rotation sensor that detects the rotation of the rotation axis AX1.

[0055] Also, point Q1 at the lower end of rotation axis AX1 shown in Fig. 11 indicates the position of the MP joint of the wearer's finger. Specifically, taking the example of an index finger, which is one finger, the wearer who wears exoskeleton robot 120 wears exoskeleton unit 120A so that the position of the MP joint of the index finger is at the position of the black circle shown at the lower end of rotation axis AX1 in Fig. 11. Also, the wearer wears exoskeleton unit 120A on the index finger so that the range from the MP joint to the tip of the index finger is aligned with the RCM mechanism link part. The RCM mechanism link part extends and contracts in accordance with the movement of the wearer bending and straightening the index finger.

[0056] The exoskeleton unit 120A also includes an RCM mechanism link section with a double parallelogram structure using parallel links. The RCM mechanism link section includes an RCM mechanism. For example, the RCM mechanism link section shown in FIG. 11 has a structure in which parallel links are combined to form two symmetrical parallelograms (also called double parallelograms). The exoskeleton unit 120A also includes a first link L1 with one end fixed to a second vertex P2 diagonally located with respect to the connection point PX of the two parallelogram structures included in the RCM mechanism link section, a second link L2 with one end connected to the other end of the first link L1 by a first ball joint J1, and a third link L3 with one end connected to the other end of the second link L2 by a second ball joint J2, and the first link L1, the second link L2, and the third link L3 form a four-joint link structure. Moreover, the exoskeleton unit 120A further includes a motor 121A, and the other end of the third link L3 is fixed to the rotation shaft of the motor 121A.

[0057] Point Q2 below the RCM mechanism link unit shown in FIG. 11 indicates the position of the rotation center of the RCM mechanism provided in the RCM mechanism link unit. Moreover, position Q2 of the rotation center of the RCM mechanism is located on the symmetrical axis of the double parallelogram. Moreover, position Q2 of the rotation center of the RCM mechanism is located between the MP joint and the PIP joint of the wearer's index finger. In this way, the exoskeleton unit 120A is provided with a rotation axis of the RCM mechanism within a predetermined range from the position of the MP joint of the finger and a rotation axis of the rotation mechanism that rotates the RCM mechanism in accordance with the adduction or abduction rotation of the finger.

[0058] Next, an RCM mechanism provided in an exoskeleton unit corresponding to one finger of the exoskeleton robot according to the first embodiment of the present disclosure will be described with reference to Fig. 12. Fig. 12 is a diagram for describing an RCM mechanism provided in an exoskeleton unit 120A corresponding to one finger of the exoskeleton robot 120 according to the first embodiment of the present disclosure. The left side of Fig. 12 shows a state in which the finger is extended. The right side of Fig. 12 shows a state in which the finger is bent.

[0059] As shown in Fig. 12, the exoskeleton unit 120A includes a four-joint link by two ball bearings that receive two ball joints J1 and J2. Specifically, the exoskeleton unit 120A includes a first link L1, one end of which is fixed to a second vertex P2 diagonally located with respect to the connection point PX of the two parallelogram structures included in the RCM mechanism link section, a second link L2, one end of which is connected to the other end of the first link L1 by the first ball joint J1, and a third link L3, one end of which is connected to the other end of the second link L2 by the second ball joint J2, and the first link L1, the second link L2, and the third link L3 form a four-joint link structure. The exoskeleton unit 120A further includes a motor 121A, and the other end of the third link L3 is fixed to the rotation shaft of the motor 121A. Moreover, the exoskeleton unit 120A is provided with a rotation axis AX1 of the degree of freedom of adduction or abduction of the fingers at a position just below the ball joint J2. Furthermore, point Q1 at the lower end of the rotation axis AX1 corresponds to the position of the MP joint of the wearer's fingers. This allows the exoskeleton robot 120 to reduce frictional resistance in the drive transmission section and improve the degree of freedom of adduction and abduction of the fingers, thereby enabling the wearer of the exoskeleton robot 120 to reproduce smoother and more natural finger movements.

[0060] Also, the position Q2 of the rotation center of the RCM mechanism is located between the MP joint and the PIP joint of the wearer's finger. In other words, the position Q2 of the rotation center of the RCM mechanism is located within a predetermined range from the MP joint of the wearer's finger. In this way, the exoskeleton unit 120A is provided with the rotation axis of the RCM mechanism and the rotation axis of the rotation mechanism that rotates the RCM mechanism in accordance with the rotation of the adduction or abduction of the finger within a predetermined range from the position of the MP joint of the finger. Here, when viewed from the side of the wearer's wrist, the exoskeleton unit 120A has the degrees of freedom of the device arranged in the order of the rotation axis AX1 of the degree of freedom of the adduction or abduction of the outer finger, and the RCM mechanism. In general, the adduction or abduction movement of the finger is performed by rotating around the rotation axis AX1 by the MP joint of the finger. Also, the movement of bending and straightening the finger is performed by rotating around the rotation axis of the RCM mechanism by the MP joint, PIP joint, and DIP joint of the finger. Also, the natural movement of the fingers can be considered to be a combination of the adduction or abduction movement of the fingers and the bending and straightening movement of the fingers. Therefore, the natural movement of the fingers can be considered to be performed in the order of the adduction or abduction movement of the fingers rotating around the rotation axis AX1 and the bending and straightening movement of the fingers rotating around the rotation axis of the RCM mechanism when viewed from the wrist side. Considering this, the exoskeleton unit 120A can guide the fingers in the same way as the original finger movement because the degrees of freedom of the device are arranged in the order of the rotation axis AX1 of the degree of freedom of the adduction or abduction of the outer fingers and the RCM mechanism when viewed from the wrist side of the wearer. In other words, the exoskeleton unit 120A can guide the adduction or abduction movement of the fingers based on the MP joints of the fingers and the bending and straightening movement of the fingers based on the MP joints, PIP joints, and DIP joints of the fingers in the same way as the original finger movement.

[0061] In this way, the information processing device 100 further includes an exoskeleton robot 120 including an RCM mechanism having a double parallelogram structure using parallel links, a rotation mechanism for rotating the RCM mechanism in accordance with the rotation of the adduction or abduction of the fingers, a holding member fixed to a first vertex P1 diagonally positioned with respect to the connection point PX of the two parallelogram structures of the RCM mechanism and holding the fingers, a first link L1 having one end fixed to a second vertex P2 diagonally positioned with respect to the connection point PX, a second link L2 having one end connected to the other end of the first link L1 by a first ball joint J1, and a third link L3 having one end connected to the other end of the second link L2 by a second ball joint J2, the first link L1, the second link L2, and the third link L3 forming a four-joint link structure. The exoskeleton robot 120 further includes a motor 121A, and the other end of the third link L3 is fixed to the rotation shaft of the motor 121A. The rotation axis of the rotation mechanism and the rotation axis of the RCM mechanism are arranged in the order of the rotation axis of the rotation mechanism and the rotation axis of the RCM mechanism from the side closer to the wrist of the wearer of the exoskeleton robot 120. The rotation axis of the rotation mechanism is located at the MP joint of the finger of the wearer of the exoskeleton robot, and the rotation axis of the RCM mechanism is located within a predetermined range from the MP joint of the finger in the direction from the MP joint of the finger of the wearer of the exoskeleton robot to the PIP joint. As a result, when viewed from the wrist side, the information processing device 100 can guide the finger in the same way as the original finger movement because the degrees of freedom of the device are arranged in the order of the rotation axis of the degree of freedom of the adduction or abduction of the finger and the rotation axis of the RCM mechanism. That is, the information processing device can guide the adduction or abduction movement of the finger based on the MP joint of the finger and the bending and straightening movement of the finger based on the MP joint, PIP joint, and DIP joint of the finger in the same way as the original finger movement.

[0062] Next, the exoskeleton robot according to the first embodiment of the present disclosure will be further described with reference to Fig. 13. Fig. 13 is a top view of the exoskeleton robot according to the first embodiment of the present disclosure. The left side of Fig. 13 is a top view of Fig. 9. The right side of Fig. 13 shows a state in which the exoskeleton unit 120A corresponding to the index finger of the right hand has been moved with the screw of the MP joint position adjustment mechanism of the exoskeleton unit 120A corresponding to the index finger of the right hand loosened.

[0063] (Storage unit 130) The storage unit 130 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 130 stores the sensor information acquired from the finger sensor 110. The storage unit 130 also stores information related to the feature amount of the fingers calculated by the estimation unit 152.

[0064] (Communication unit 140) The communication unit 140 is realized by, for example, a network interface card (NIC) etc. The communication unit 140 is connected to a network by wire or wirelessly, and transmits and receives information to and from, for example, the application server 10 and the terminal device 20.

[0065] (Control unit 150) The control unit 150 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 150 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0066] 3, the control unit 150 has a sensor information processing unit 151, an estimation unit 152, a drive control unit 153, and a transmission unit 154, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 150 is not limited to the configuration shown in FIG. 3, and may be other configurations as long as they perform the information processing described below.

[0067] (Sensor information processing unit 151) The sensor information processing unit 151 acquires raw data sensed by the hand sensor 110 from the hand sensor 110. The sensor information processing unit 151 performs processing such as sampling of the signal detected by the hand sensor 110, noise reduction, filter application, and converting the force into a voltage.

[0068] (Estimation part 152) The estimation unit 152 acquires sensor information of the finger sensor 110 from the sensor information processing unit 151. Then, when the estimation unit 152 acquires the sensor information, it estimates information on the feature amounts of the fingers based on the acquired sensor information. Specifically, the estimation unit 152 estimates information on the feature amounts of the fingers, which are the angles of the finger joints, the average angular velocity of the finger joints, the maximum angular velocity, and the range of motion of the finger joints. Then, when the estimation unit 152 estimates the information on the feature amounts of the fingers, it stores the estimated information on the feature amounts of the fingers in the storage unit 130.

[0069] Specifically, the estimation unit 152 acquires sensor information from the finger sensor 110, which is related to the intensity of light reflected by the finger after light is irradiated onto the finger from a photoreflector provided in the finger sensor 110. For example, the estimation unit 152 irradiates infrared light from a photoreflector attached on the extension of the upper surface of the middle phalangeal joint, and acquires sensor information related to the intensity of light reflected by the surface of the skin of the distal phalangeal joint and returned. Alternatively, the estimation unit 152 acquires sensor information related to the distance between the sensor and the skin of the distal phalangeal joint, which is measured based on the intensity of light. Next, the estimation unit 152 calculates the angle θ2 of the DIP joint based on the acquired sensor information. Next, the estimation unit 152 estimates the angular velocity of the DIP joint by differentiating the angle θ2 of the DIP joint with respect to time. In addition, the estimation unit 152 estimates the range of motion of the DIP joint based on the maximum and minimum values ​​of the angle θ2 within the range in which the DIP joint moved during sensing. In this manner, the estimation unit (estimation unit 152) estimates the posture of the fingers based on the angles of the joints detected by the detection unit (photoreflector).

[0070] The estimation unit 152 also irradiates infrared light from a photoreflector attached on the extension of the upper surface of the middle joint, and acquires sensor information related to the intensity of the light reflected and returned from the surface of the skin of the proximal joint. Alternatively, the estimation unit 152 acquires sensor information related to the distance between the sensor and the skin of the proximal joint measured based on the intensity of the light. Next, the estimation unit 152 calculates the angle θ1 of the PIP joint based on the acquired sensor information. Next, the estimation unit 152 estimates the angular velocity of the PIP joint by differentiating the angle θ1 of the PIP joint with respect to time. Furthermore, the estimation unit 152 estimates the range of motion of the PIP joint by integrating the angle θ1 of the PIP joint with respect to time.

[0071] The estimation unit 152 also acquires sensor information related to the posture of the back of the hand (or the palm) detected by a 9-axis posture sensor attached to the back of the hand. Next, the estimation unit 152 estimates the absolute angle α0 of the palm with respect to the horizontal plane based on the acquired sensor information.

[0072] The estimation unit 152 also acquires sensor information related to the orientation of the middle joint detected by a 9-axis orientation sensor attached to the middle joint. Next, the estimation unit 152 estimates the absolute angle α1 of the middle joint with respect to the horizontal plane based on the acquired sensor information.

[0073] The estimation unit 152 also calculates the MP joint angle θ0 based on the estimated palm absolute angle α0 and middle joint absolute angle α1, the calculated PIP joint angle θ1, and the relational expression "α0-θ0-θ1=α1" described in Fig. 7. The estimation unit 152 then estimates the angular velocity of the MP joint by differentiating the MP joint angle θ0 with respect to time. The estimation unit 152 also estimates the range of motion of the MP joint by integrating the MP joint angle θ0 with respect to time.

[0074] The estimation unit 152 may estimate the posture of the fingers based on the state of the exoskeleton robot and the angles of the joints detected by the detection unit (photoreflector). This allows the information processing device 100 to more appropriately estimate the posture of the fingers.

[0075] (Drive control unit 153) The drive control unit 153 refers to the storage unit 130 and acquires information related to the feature amount of the fingers estimated by the estimation unit 152. Then, when the drive control unit 153 acquires the information related to the feature amount of the fingers, the drive control unit 153 generates a command for the exoskeleton robot 120 based on the acquired information. Then, when the drive control unit 153 generates a command for the exoskeleton robot 120, the drive control unit 153 controls the drive of the exoskeleton robot 120 based on the generated command.

[0076] (Transmitter 154) The transmission unit 154 transmits information on the feature amounts of the fingers estimated by the estimation unit 152 to the application server 10. Specifically, the transmission unit 154 transmits information on the feature amounts of the fingers estimated by the estimation unit 152 in response to a request from the application server 10. For example, the transmission unit 154 refers to the storage unit 130 and transmits information on the feature amounts of the fingers, which are the angles, average angular velocities, maximum angular velocities, and ranges of motion of the DIP joints, PIP joints, or MP joints of the fingers.

[0077] [1.3. Example of operation of information processing system] Next, an information processing procedure according to the first embodiment of the present disclosure will be described with reference to FIG. 14. FIG. 14 is a flowchart showing an information processing procedure according to the first embodiment of the present disclosure. As shown in FIG. 14, the information processing device 100 acquires sensor information of the finger sensor 110 (step S101). Next, the information processing device 100 calculates the feature amount of the finger based on the sensor information acquired from the finger sensor 110 (step S102). Next, when the information processing device 100 calculates the feature amount of the finger, it generates command information for the exoskeleton robot 120 (step S103). Next, when the information processing device 100 generates the command information for the exoskeleton robot 120, it controls the operation of the exoskeleton robot 120 based on the generated command information (step S104).

[0078] [1.4. Modifications] [1.4.1. Hand sensor] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to FIG. 15. FIG. 15 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. The finger sensor 111 shown in FIG. 15 differs from the finger sensor 110 shown in FIG. 4 in that a 9-axis orientation sensor 110D for measuring the wrist orientation is further provided on the forearm. By using the finger sensor 111, the relative angle between the absolute angle of the back of the hand detected by the 9-axis orientation sensor 110C for measuring the reference orientation of the back of the hand and the absolute angle of the wrist detected by the 9-axis orientation sensor 110D for measuring the wrist orientation can be calculated as the tilt of the wrist.

[0079] In this way, the information processing device 100 further includes a third attitude sensor (9-axis attitude sensor 110D) for detecting the inclination of the wrist with respect to the direction of gravity. The third attitude sensor is a 9-axis sensor that detects the acceleration in three axial directions, the angular velocity in three axial directions, and the orientation in three axial directions. This allows the information processing device 100 to appropriately detect the inclination of the wrist with respect to the direction of gravity. The estimation unit calculates the inclination of the wrist with respect to the direction of gravity with respect to the back of the hand as the inclination angle of the wrist, based on the inclination of the wrist with respect to the direction of gravity detected by the third attitude sensor and the inclination of the back of the hand with respect to the direction of gravity detected by the second attitude sensor. This allows the information processing device 100 to estimate the posture of the wrist, thereby further improving the estimation accuracy of the posture of the fingers.

[0080] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to FIG. 16. FIG. 16 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. The finger sensor 112 shown in FIG. 16 is different from the finger sensor 110 shown in FIG. 4 in that it further includes a 9-axis orientation sensor 110E for measuring the curvature of the palm on the extension of the base bone of the ring finger and little finger on the back of the hand. By using the finger sensor 112, the relative angle between the absolute angle of the back of the hand detected by the 9-axis orientation sensor 110C for measuring the reference orientation of the back of the hand and the absolute angle of the curvature of the palm detected by the 9-axis orientation sensor 110E for measuring the curvature of the palm can be calculated as the degree of curvature of the palm. In this case, when calculating the extension and flexion angles of each joint of the ring finger and little finger, the angle of the 9-axis orientation sensor 110E for measuring the curvature of the palm with respect to the horizontal plane can be used as the above-mentioned α0, thereby improving the accuracy of calculation of the extension and flexion angles while curving the palm.

[0081] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to FIG. 17. FIG. 17 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. The finger sensor 113 shown in FIG. 17 differs from the finger sensor 110 shown in FIG. 4 in that the second sensor unit 110B is attached to the proximal joint of each finger. When the finger sensor 113 is used, the angle of the DIP joint cannot be measured, but it allows measurement with less constraint. In addition, when the finger sensor 113 is used, the angle of the MP joint of each finger can be directly calculated from the relative angle between the 9-axis posture sensor of the second sensor unit 110B and the 9-axis posture sensor 110C for measuring the reference posture of the back of the hand.

[0082] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to FIG. 18. FIG. 18 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. The finger sensor 114 shown in FIG. 18 differs from the finger sensor 110 shown in FIG. 4 in that in addition to the first sensor unit 110A, a second sensor unit 110B is attached to the proximal joint of each finger. When the finger sensor 114 is used, the angle of the MP joint of each finger can be directly calculated from the relative angle between the 9-axis posture sensor of the second sensor unit 110B and the 9-axis posture sensor 110C for measuring the reference posture of the back of the hand.

[0083] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. Finger sensor 115 shown in Fig. 19 differs from finger sensor 110 shown in Fig. 4 in that some sensors are replaced with bending sensors or strain sensors arranged to sandwich the joint to be measured. Use of finger sensor 115 can improve the measurement accuracy of the angle of the joint to be measured.

[0084] Next, a finger sensor according to a modified example of the first embodiment of the present disclosure will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of a finger sensor according to a modified example of the first embodiment of the present disclosure. The finger sensor 116 shown in Fig. 20 differs from the finger sensor 110 shown in Fig. 4 in that it does not use a 9-axis attitude sensor and a photoreflector is also installed on the MP joint. By using the finger sensor 116, the angles of all the joints of each finger can be directly calculated by measurement using the photoreflector.

[0085] [1.4.2. Exoskeleton robots] Next, an exoskeleton robot according to a modified example of the first embodiment of the present disclosure will be described with reference to Fig. 21. Fig. 21 is a perspective view of an exoskeleton robot according to a modified example of the first embodiment of the present disclosure. The exoskeleton robot shown in Fig. 21 differs from the exoskeleton robot 120 shown in Fig. 9 in that an exoskeleton unit corresponding to a thumb is added.

[0086] Next, the exoskeleton robot according to a modified example of the first embodiment of the present disclosure will be further described with reference to Fig. 22. Fig. 22 is a side view of the exoskeleton robot according to a modified example of the first embodiment of the present disclosure. Fig. 22 is a side view of the exoskeleton robot to which an exoskeleton unit corresponding to the thumb shown in Fig. 21 is added.

[0087] Next, the exoskeleton robot according to the modified example of the first embodiment of the present disclosure will be further described with reference to Fig. 23. Fig. 23 is a top view of the exoskeleton robot according to the modified example of the first embodiment of the present disclosure. Fig. 23 is a top view of the exoskeleton robot to which an exoskeleton unit corresponding to the thumb shown in Fig. 21 is added.

[0088] Next, the exoskeleton robot according to the modified example of the first embodiment of the present disclosure will be further described with reference to Fig. 24. Fig. 24 is a bottom view of the exoskeleton robot according to the modified example of the first embodiment of the present disclosure. Fig. 24 is a bottom view of the exoskeleton robot to which an exoskeleton unit corresponding to the thumb shown in Fig. 21 is added.

[0089] [2. Second embodiment] [2.1. Overview of information processing system] Next, an overview of information processing according to the second embodiment of the present disclosure will be described with reference to FIG. 25. FIG. 25 is a diagram showing an example of information processing according to the second embodiment of the present disclosure. In the example shown in FIG. 25, an information processing device 200 is different from the information processing device 100 of FIG. 1 in that an object sensor 210 embedded in an object O1 is provided instead of a finger sensor 110. Here, the object O1 refers to an object of a motion by fingers. For example, when the motion by fingers is playing a piano, the object O1 of the motion by fingers corresponds to the keys of the piano.

[0090] Generally, sensors attached to the exoskeleton robot itself are used to control the exoskeleton robot, but in cases such as for the purpose of assisting skill development, it may be possible to embed sensors in the object of finger movement. Also, when using an exoskeleton robot to control an object through the wearer's fingers, as shown in Figure 27 described later, a model of the wearer's fingers is placed between the output of the exoskeleton robot and the object, making it difficult to control correctly.

[0091] Therefore, in the information processing system according to the second embodiment of the present disclosure, an information processing system 2 is proposed that uses sensor information from a sensor embedded in an object of finger movement to control an exoskeleton robot more accurately and safely. The information processing system 2 attempts to achieve more accurate control by feeding back information from the sensor embedded in the object to the exoskeleton robot. Specifically, stable control is performed by rotating the control loop shown in FIG. 25 at a high speed of 100 Hz or more. Note that some control loops, such as the control loop in the exoskeleton robot 220, may be made faster. Note that the information processing system 2 differs from the information processing system 1 shown in FIG. 2 in that it includes an information processing device 200 instead of the information processing device 100, but is otherwise the same.

[0092] In the example shown in FIG. 25, the estimation unit 252 of the information processing device 200 acquires sensor information from the object sensor 210. For example, the estimation unit 252 acquires sensor information such as the displacement of the deformation of the object O1 caused by the finger of the wearer wearing the exoskeleton robot 220 pressing the object O1, the speed of the deformation, and the force applied to the object. The estimation unit 252 also acquires model information of the object O1. For example, if the object O1 is a piano key, the model information of the object O1 refers to information about a model having the shape of a piano key. Next, when the estimation unit 252 acquires the sensor information and the model information, the estimation unit 252 estimates the state of the object O1 based on the acquired sensor information and the model information. Next, when the estimation unit 252 estimates the state of the object O1, the estimation unit 252 outputs information about the estimated state of the object O1 to the drive control unit 253.

[0093] Furthermore, the estimation unit 252 of the information processing device 200 acquires sensor information of the exoskeleton robot 120 from the sensor 122 of the exoskeleton robot 120. Next, upon acquiring the sensor information of the exoskeleton robot 120, the estimation unit 252 estimates a state of the exoskeleton robot 120 based on the acquired sensor information. Next, upon estimating the state of the exoskeleton robot 120, the estimation unit 252 outputs information relating to the estimated state of the exoskeleton robot 120 to the drive control unit 253.

[0094] The drive control unit 253 of the information processing device 200 acquires information on the state of the object O1 and information on the state of the exoskeleton robot 120 from the estimation unit 252. Next, upon acquiring the information on the state of the object O1 and information on the state of the exoskeleton robot 120, the drive control unit 253 controls the driving of the drive unit 121 (e.g., a motor) of the exoskeleton robot 120 based on the acquired information on the state of the object O1 and information on the state of the exoskeleton robot 120.

[0095] [2.2. Example of information processing device configuration] Next, the configuration of an information processing device according to a second embodiment of the present disclosure will be described with reference to FIG. 26. FIG. 26 is a diagram showing a configuration example of an information processing device according to a second embodiment of the present disclosure. As shown in FIG. 26, the information processing device 200 according to the second embodiment includes an exoskeleton robot 120, a storage unit 130, and a communication unit 140, similar to the information processing device 100 according to the first embodiment. In addition, the information processing device 200 differs from the information processing device 100 in that it includes an object sensor 210 and a control unit 250. In the following, the description of the parts that overlap with the information processing device 100 will be omitted as appropriate.

[0096] (Object sensor 210) The object sensor 210 is a sensor embedded in the object O1 to be operated by the fingers. For example, the object sensor 210 is realized by a contact sensor mounted inside the object. For example, the object sensor 210 detects the displacement of the deformation of the object O1 caused by the finger of the wearer wearing the exoskeleton robot 220 pressing down on the object O1, the speed of the deformation, the force applied to the object, and the like.

[0097] (Control unit 250) The control unit 250 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 250 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0098] 26, the control unit 250 has a sensor information processing unit 251, an estimation unit 252, a drive control unit 253, and a transmission unit 154, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 250 is not limited to the configuration shown in FIG. 26, and may be other configurations as long as they perform the information processing described below.

[0099] (Sensor information processing unit 251) The sensor information processing unit 251 acquires raw data sensed by the object sensor 210 from the object sensor 210. The sensor information processing unit 251 performs processing such as sampling of the signal detected by the object sensor 210, noise reduction, filter application, and converting the force into a voltage.

[0100] (Estimation part 252) The estimation unit 252 acquires sensor information detected by the object sensor 210 from the sensor information processing unit 251. For example, the estimation unit 252 acquires sensor information such as the displacement of the deformation of the object O1 caused by the finger of the wearer wearing the exoskeleton robot 220 pressing the object O1, the speed of the deformation, and the force applied to the object. The estimation unit 252 also acquires model information of the object O1. Next, when the estimation unit 252 acquires the sensor information and the model information, the estimation unit 252 estimates the state of the object O1 based on the acquired sensor information and model information. Next, when the estimation unit 252 estimates the state of the object O1, the estimation unit 252 outputs information on the estimated state of the object O1 to the drive control unit 253.

[0101] Furthermore, the estimation unit 252 acquires sensor information of the exoskeleton robot 120 from the sensor 122 of the exoskeleton robot 120. Next, when the estimation unit 252 acquires the sensor information of the exoskeleton robot 120, it estimates the state of the exoskeleton robot 120 based on the acquired sensor information. For example, the estimation unit 252 estimates information such as the force and speed applied by the exoskeleton robot 120 and the force and speed exerted by the wearer of the exoskeleton robot 120. Next, when the estimation unit 252 estimates the state of the exoskeleton robot 120, it outputs information on the estimated state of the exoskeleton robot 120 to the drive control unit 253.

[0102] (Drive control unit 253) The drive control unit 253 acquires information on the state of the object O1 and information on the state of the exoskeleton robot 120 from the estimation unit 252. Next, when the drive control unit 253 acquires the information on the state of the object O1 and the information on the state of the exoskeleton robot 120, the drive control unit 253 recalculates the output of the motor 121A with respect to the target force and position based on the acquired information on the state of the object O1 and information on the state of the exoskeleton robot 120. Next, when the drive control unit 253 recalculates the output of the motor 121A, it controls the driving of the motor 121A based on the recalculated output.

[0103] [2.3. Example of operation of information processing system] Next, the flow of information processing according to the second embodiment of the present disclosure will be conceptually described with reference to FIG. 27. FIG. 27 is a diagram for conceptually describing the flow of information processing according to the second embodiment of the present disclosure. As shown in FIG. 27, between an object of operation such as a piano keyboard according to the second embodiment and the wearer's fingers, a force is physically applied to or from the wearer's fingers. Similarly, a force is physically applied to or from the wearer's fingers and the exoskeleton robot.

[0104] The estimation unit 252 also estimates the state of the object, such as the force applied to the object and the position of the object, based on the sensor information of the object sensor embedded in the object and the model information of the object. The estimation unit 252 also estimates the state of the exoskeleton robot, such as the angle and force of the joints of the exoskeleton robot, based on the sensor information of the exoskeleton robot.

[0105] Next, the drive control unit 253 controls the drive of the actuator of the exoskeleton robot based on the target state of the exoskeleton robot acquired in advance, and the state of the object and the state of the exoskeleton robot estimated by the estimation unit 252. For example, the drive control unit 253 controls the drive of the motor of the exoskeleton robot. The drive control unit 253 also controls the drive of the link portion of the exoskeleton robot by controlling the drive of the motor.

[0106] Next, an information processing procedure according to the second embodiment of the present disclosure will be described with reference to FIG. 28. FIG. 28 is a flowchart showing an information processing procedure according to the second embodiment of the present disclosure. As shown in FIG. 28, the estimation unit 252 acquires sensor information of the object sensor 210 (step S201). Then, the estimation unit 252 acquires model information of the object O1 (step S202). Then, the estimation unit 252 estimates information on the state of the object O1 based on the sensor information and the model information (step S203). Then, the estimation unit 252 outputs information on the state of the object O1 to the drive control unit 253 (step S204).

[0107] Next, the information processing procedure according to the second embodiment of the present disclosure will be further described with reference to FIG. 29. FIG. 29 is a flowchart showing the information processing procedure according to the second embodiment of the present disclosure. As shown in FIG. 29, the estimation unit 252 acquires sensor information of the exoskeleton robot 220 (step S301). Next, the estimation unit 252 estimates information related to the state of the exoskeleton robot 220 based on the sensor information (step S302). Next, the estimation unit 252 outputs the information related to the state of the exoskeleton robot 220 to the drive control unit 253 (step S303).

[0108] Next, the information processing procedure according to the second embodiment of the present disclosure will be further described with reference to FIG. 30. FIG. 30 is a flowchart showing the information processing procedure according to the second embodiment of the present disclosure. As shown in FIG. 30, the drive control unit 253 acquires information on the state of the object O1 and information on the state of the exoskeleton robot 220 from the estimation unit 252 (step S401). Next, the drive control unit 253 controls the operation of the exoskeleton robot 220 based on the information on the state of the object O1, information on the state of the exoskeleton robot 220, and information on the target state of the exoskeleton robot 220 (step S402).

[0109] [3.Other] Next, a screen example when the processing result according to the embodiment is displayed on the terminal device of the user will be described with reference to FIG. 31. FIG. 31 is a diagram showing an example of a screen when the processing result according to the embodiment is displayed on the terminal device of the user. In the example shown in FIG. 31, the terminal device 20 displays information on the posture and position of the user's fingers estimated by the information processing device 100 and information on the posture and position of the model fingers in a comparable manner. In FIG. 31, the image of the hand indicated by "You" on the left side of the screen of the terminal device 20 corresponds to information on the posture and position of the user's fingers. Also, the image of the hand indicated by "Teacher" on the right side of the screen of the terminal device 20 corresponds to information on the posture and position of the model fingers. Also, the terminal device 20 may display information on the posture and position of the user's fingers estimated by the information processing device 100 and information on the past posture and position of the user's fingers in a comparable manner. For example, the terminal device 20 displays an image on the left side of the screen related to the posture and position of the user's fingers estimated by the information processing device 100, and displays an image on the right side of the screen related to the past posture and position of the user's fingers. Furthermore, the terminal device 20 may display a video of the fingers in motion for comparison, or may display a still image of the fingers at a certain moment for comparison. In this manner, the terminal device 20 displays information on the posture and position of the user's fingers estimated by the information processing device 100.

[0110] Next, an example of a screen when the processing result according to the embodiment is displayed on the terminal device of the user will be described with reference to FIG. 32. FIG. 32 is a diagram showing an example of a screen when the processing result according to the embodiment is displayed on the terminal device of the user. In the example shown in FIG. 32, for example, the terminal device 20 displays a difference between the motion of the user's fingers estimated by the information processing device 100 and the motion of the model finger by a moving image. In addition, the terminal device 20 displays a three-dimensional feature amount such as the speed of the tip of the user's finger estimated by the information processing device 100 and a three-dimensional feature amount such as the speed of the tip of the model finger in a manner that allows comparison. In addition, the terminal device 20 may display a difference between the motion of the user's fingers estimated by the information processing device 100 and the past motion of the user's fingers by a moving image. For example, the terminal device 20 displays a three-dimensional feature amount such as the speed of the tip of the user's finger estimated by the information processing device 100 and a three-dimensional feature amount such as the speed of the tip of the user's finger in a manner that allows comparison. In addition, the terminal device 20 may be AR (Augmented Reality) glasses or HMD (Head Mounted Display). A user wears the terminal device 20, which is an AR glass or an HMD. The terminal device 20 then displays images of model finger movements and past finger movements of the user by superimposing them on the user's actual finger movements using AR. This allows the terminal device 20 to present information to the user more specifically.

[0111] [4. Effects] As described above, the information processing device according to the first embodiment, the second embodiment, and the modified example includes a detection unit and an estimation unit. The detection unit detects angles of at least some of the joints of the fingers in a non-contact manner. The estimation unit estimates the posture of the fingers based on the angles of the joints detected by the detection unit.

[0112] As a result, the information processing device can detect the angles of the finger joints without interfering with the fine sensation of the fingertips or the dexterity of the fingers, since the information processing device detects without contact. Therefore, the information processing device can appropriately measure the movements of the fingers.

[0113] The detection unit also includes a non-contact sensor, detects the distance from the non-contact sensor to the finger, which changes depending on the angle of the joint, and detects the angle of the joint based on the detected distance. The non-contact sensor also includes a light-emitting element that emits light and a light-receiving element that detects reflected light. The light-emitting element emits light to the joint of the finger. The light-receiving element detects the light reflected by the joint of the finger. The detection unit detects the distance to the finger based on the intensity of the reflected light detected by the light-receiving element.

[0114] This enables the information processing device to indirectly (i.e., non-contact) detect the angle of a joint by detecting the distance from a non-contact sensor to the finger, instead of attaching a sensor to the joint to directly detect the angle of the joint.

[0115] The non-contact sensor is provided on at least one of the middle phalanges of the index finger, middle finger, ring finger, and little finger. The light emitting element irradiates light to at least one of the distal phalanges and the proximal phalanges of the fingers. The light receiving element detects light reflected by at least one of the distal phalanges and the proximal phalanges of the fingers. The detection unit calculates the relative angle of the distal phalange of the fingers to the middle phalange as the angle of the DIP joint of the fingers based on the distance between the distal phalange of the fingers and the non-contact sensor calculated based on the intensity of the reflected light. The estimation unit calculates the relative angle of the proximal phalange of the fingers to the middle phalange as the angle of the PIP joint of the fingers, and estimates the posture of the fingers based on the angle of the DIP joint and the angle of the PIP joint of the fingers.

[0116] This enables the information processing device to detect the angles of the DIP joints and the PIP joints of the fingers in a non-contact manner, without attaching sensors to the distal and proximal phalanges of the fingers.

[0117] The information processing device further includes a first posture sensor for detecting the inclination of the middle phalanges of the fingers with respect to the direction of gravity. The estimation unit calculates the angle of the MP joints of the fingers based on the angle of the PIP joints of the fingers and the inclination of the middle phalanges of the fingers with respect to the direction of gravity detected by the first posture sensor. The information processing device further includes a second posture sensor for detecting the inclination of the back of the hand with respect to the direction of gravity. The estimation unit calculates the relative angle of the proximal phalanges of the fingers with respect to the back of the hand as the angle of the MP joint based on the inclination of the back of the hand with respect to the direction of gravity detected by the second posture sensor.

[0118] This enables the information processing device to calculate the angle of the MP joint in a non-contact manner without attaching a sensor to the MP joint.

[0119] The second posture sensor is provided on the back of the hand, on the base bones of the index finger and middle finger, in a region between the MP joint and the wrist.

[0120] This enables the information processing device to detect the inclination of the back of the hand with respect to the direction of gravity without being affected by the curvature of the palm or the movement of the thumb.

[0121] The information processing device further includes a third posture sensor for detecting a tilt of the wrist with respect to the direction of gravity. The estimation unit calculates the tilt of the wrist with respect to the direction of gravity with respect to the back of the hand as a tilt angle of the wrist based on the tilt of the wrist with respect to the direction of gravity detected by the third posture sensor and the tilt of the back of the hand with respect to the direction of gravity detected by the second posture sensor.

[0122] This enables the information processing device to estimate the wrist posture, thereby further improving the accuracy of estimating the posture of the fingers.

[0123] The first posture sensor detects the inclination of the adduction or abduction of the fingers relative to the direction of gravity. The second posture sensor detects the inclination of the adduction or abduction of the palm relative to the direction of gravity. The estimation unit calculates information about the relative angle of the adduction or abduction of the fingers with respect to the dorsum of the hand based on the inclination of the adduction or abduction of the fingers detected by the first posture sensor relative to the direction of gravity and the inclination of the adduction or abduction of the palm detected by the second posture sensor relative to the direction of gravity.

[0124] This enables the information processing device to estimate the angle of adduction or abduction of the fingers with relatively low constraints.

[0125] The first attitude sensor is a nine-axis sensor, the second attitude sensor is a nine-axis sensor, and the third attitude sensor is a nine-axis sensor.

[0126] This enables the information processing device to appropriately detect the inclination of the middle joint of the finger relative to the direction of gravity, the inclination of the back of the hand relative to the direction of gravity, and the inclination of the wrist relative to the direction of gravity.

[0127] The information processing device further includes an exoskeleton robot including an RCM (Remote Center of Motion) mechanism having a double parallelogram structure using parallel links, a rotation mechanism that rotates the RCM mechanism in accordance with the rotation of the fingers inward or outward, a holding member that is fixed to a first vertex diagonally opposite to a connection point of the two parallelogram structures included in the RCM mechanism and holds the fingers, a first link having one end fixed to a second vertex diagonally opposite to the connection point, a second link having one end connected to the other end of the first link by a first ball joint, and a third link having one end connected to the other end of the second link by a second ball joint, the first link, the second link, and the third link forming a four-joint link structure. The exoskeleton robot further includes a motor, and the other end of the third link is fixed to the rotation shaft of the motor.

[0128] This allows the information processing device to reduce frictional resistance in the drive transmission unit of the exoskeleton robot and improve the degree of freedom of adduction and abduction of the fingers, thereby reproducing smoother and more natural finger movements for the wearer of the exoskeleton robot. Therefore, the information processing device can appropriately measure the finger movements and then appropriately reproduce the measured finger movements for the user.

[0129] The rotation axis of the rotation mechanism and the rotation axis of the RCM mechanism are arranged in this order from the side closer to the wrist of the person wearing the exoskeleton robot. The rotation axis of the rotation mechanism is located at the MP joint of the finger of the person wearing the exoskeleton robot, and the rotation axis of the RCM mechanism is located within a predetermined range from the MP joint of the finger in the direction facing the PIP joint from the MP joint of the finger of the person wearing the exoskeleton robot.

[0130] As a result, when viewed from the wrist side, the information processing device has the degrees of freedom of the device arranged in the order of the rotation axis of the degree of freedom of the adduction or abduction of the fingers, and the RCM mechanism, so that the information processing device can guide the fingers in the same way as the natural movement of the fingers. In other words, the information processing device can guide the adduction or abduction movement of the fingers based on the MP joints of the fingers, and the bending and straightening movement of the fingers based on the MP joints, PIP joints, and DIP joints of the fingers in the same way as the natural movement of the fingers.

[0131] In addition, the estimation unit estimates the posture of the fingers based on the state of the exoskeleton robot and the angles of the joints detected by the detection unit.

[0132] This allows the information processing device to more appropriately estimate the posture of the hand and fingers.

[0133] [5. Hardware configuration] An information device such as the information processing device 100 according to the above-described embodiment or modification is realized by a computer 1000 having a configuration as shown in FIG. 33, for example. FIG. 33 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of an information processing device such as the information processing device 100. The information processing device 100 according to the above-described embodiment or modification will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0134] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0135] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0136] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure or a modified example thereof, which is an example of program data 1350.

[0137] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0138] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0139] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment or its modification, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 150, etc. Also, the information processing program according to the embodiment or its modification of the present disclosure and data in the storage unit 130 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1350 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0140] The present technology can also be configured as follows. (1) A detection unit that detects the angle of at least a part of the joints of the finger in a non-contact manner; an estimation unit that estimates a posture of the hand and fingers based on the angles of the joints detected by the detection unit; An information processing device comprising: (2) the detection unit includes a non-contact sensor, detects a distance from the non-contact sensor to the finger, which changes depending on an angle of the joint, and detects the angle of the joint based on the detected distance. The information processing device according to (1). (3) The non-contact sensor includes a light-emitting element that emits light and a light-receiving element that detects reflected light of the light, The light emitting element irradiates light onto the joint of the finger, The light receiving element detects the light reflected by the joint of the finger, The detection unit is detecting a distance to the finger based on the intensity of the reflected light detected by the light receiving element; The information processing device according to (2). (4) The non-contact sensor is provided on the middle joint of at least one of the index finger, the middle finger, the ring finger, and the little finger, The light emitting device irradiates light to at least one of a distal phalanx and a proximal phalanx of the finger, The light receiving element detects the light reflected by at least one of a distal phalanx and a proximal phalanx of the finger, The detection unit is calculating a relative angle of the distal phalange of the finger with respect to the middle phalange of the finger as an angle of the DIP joint of the finger based on a distance between the distal phalange of the finger and the non-contact sensor calculated based on the intensity of the reflected light; The estimation unit is A relative angle of the proximal phalanx of the finger with respect to the middle phalanx of the finger is calculated as an angle of the PIP joint of the finger, and a posture of the finger is estimated based on the angle of the DIP joint and the angle of the PIP joint of the finger. The information processing device according to (3). (5) A first posture sensor is further provided for detecting an inclination of the middle joint of the finger with respect to a direction of gravity, The estimation unit is Calculating an angle of an MP joint of the finger based on an angle of the PIP joint of the finger and an inclination of the middle joint of the finger with respect to the direction of gravity detected by the first posture sensor. The information processing device according to (4). (6) Further comprising a second posture sensor for detecting an inclination of the back of the hand with respect to the direction of gravity; The estimation unit is calculating a relative angle of the proximal phalanges of the fingers with respect to the back of the hand as an angle of the MP joint based on an inclination of the back of the hand with respect to the direction of gravity detected by the second posture sensor; The information processing device according to (5). (7) The second posture sensor is provided on the back of the hand, on the base bones of the index finger and middle finger, in an area between the MP joint and the wrist. The information processing device according to (6). (8) Further comprising a third attitude sensor for detecting an inclination of the wrist with respect to the direction of gravity; The estimation unit is calculating a tilt of the wrist with respect to the back of the hand with respect to the direction of gravity as a tilt angle of the wrist based on a tilt of the wrist with respect to the direction of gravity detected by the third posture sensor and a tilt of the back of the hand with respect to the direction of gravity detected by the second posture sensor; The information processing device according to (6). (9) the first posture sensor detects a tilt of the adduction or abduction of the finger with respect to the direction of gravity; The second posture sensor detects a tilt of the palm of the hand with respect to the direction of gravity when the palm of the hand is inwardly or outwardly moved, The estimation unit is calculating information regarding a relative angle of the adduction or abduction of the fingers with respect to the dorsum of the hand based on an inclination of the adduction or abduction of the fingers with respect to the direction of gravity detected by the first posture sensor and an inclination of the adduction or abduction of the palm with respect to the direction of gravity detected by the second posture sensor; The information processing device according to (6). (10) The first attitude sensor is a 9-axis sensor. The information processing device according to any one of (5) to (9). (11) The second attitude sensor is a 9-axis sensor. The information processing device according to any one of (6) to (9). (12) The third attitude sensor is a 9-axis sensor. The information processing device according to (8). (13) A double parallelogram RCM (Remote Center of Motion) mechanism using parallel links. A rotation mechanism that rotates the RCM mechanism in accordance with the rotation of the adduction or abduction of the fingers; A holding member that is fixed to a first vertex diagonally opposite to a connection point of two parallelogram structures of the RCM mechanism and holds a finger; a first link having one end fixed to a second vertex diagonally opposite to the connecting point; a second link having one end connected to the other end of the first link by a first ball joint; a third link having one end connected to the other end of the second link by a second ball joint; Equipped with The first link, the second link, and the third link form a four-section link structure. Further comprising an exoskeleton robot, The information processing device according to any one of (1) to (12). (14) The exoskeleton robot further comprises a motor; The other end of the third link is fixed to a rotating shaft of the motor. The information processing device according to (13). (15) a rotation axis of the rotation mechanism and a rotation axis of the RCM mechanism are arranged in the order of the rotation axis of the rotation mechanism and the rotation axis of the RCM mechanism from the side closer to the wrist of the wearer of the exoskeleton robot; The information processing device according to (13) or (14). (16) The rotation axis of the rotation mechanism is located at the MP joint of the finger of the wearer of the exoskeleton robot, The rotation axis of the RCM mechanism is located within a predetermined range from the MP joint of the finger of the wearer of the exoskeleton robot in a direction from the MP joint of the finger to the PIP joint. The information processing device according to any one of (13) to (15). (17) The estimation unit estimates a posture of the hand and fingers based on a state of the exoskeleton robot and an angle of the joint detected by the detection unit. The information processing device according to any one of (13) to (16). (18) Detecting the angle of at least a part of the joints of the finger in a non-contact manner; The posture of the hand and fingers is estimated based on the detected angles of the joints. An information processing method comprising the steps of: (19) A detection process for detecting angles of at least some joints of a finger in a non-contact manner; an estimation process of estimating a posture of the hand and fingers based on the angles of the joints detected in the detection process; An information processing program for causing a computer to execute the above. [Explanation of symbols]

[0141] 1. Information Processing Systems 10 Application Server 20 Terminal Equipment 100 Information processing device 110 Hand sensor 120 Exoskeleton Robot 121 Drive unit 122 Sensors 130 Storage section 140 Communications Department 150 Control section 151 Sensor information processing unit 152 Estimation Department 153 Drive control unit 154 Transmitter

Claims

1. A detection unit including a non-contact sensor, detecting a distance from the non-contact sensor to a finger, the distance varying according to an angle of at least a part of a joint of the finger, and detecting an angle of the joint in a non-contact manner based on the detected distance; an estimation unit that estimates a posture of the hand and fingers based on the angles of the joints detected by the detection unit; Equipped with The non-contact sensor includes a light-emitting element that emits light and a light-receiving element that detects reflected light of the light, The light emitting element irradiates light onto the joint of the finger, The light receiving element detects the light reflected by the joint of the finger, The detection unit is Detecting a distance to the finger based on the intensity of the reflected light detected by the light receiving element; The non-contact sensor is provided on a middle joint of at least one of an index finger, a middle finger, a ring finger, and a little finger, The light emitting device irradiates light to at least one of a distal phalanx and a proximal phalanx of the finger, The light receiving element detects the light reflected by at least one of a distal phalanx and a proximal phalanx of the finger, The detection unit is calculating a relative angle of the distal phalange of the finger with respect to the middle phalange of the finger as an angle of a DIP joint of the finger based on a distance between the distal phalange of the finger and the non-contact sensor calculated based on the intensity of the reflected light; The estimation unit is A relative angle of the proximal phalanx of the finger with respect to the middle phalanx of the finger is calculated as an angle of the PIP joint of the finger, and a posture of the finger is estimated based on the angle of the DIP joint of the finger and the angle of the PIP joint of the finger. Information processing device.

2. A first posture sensor is further provided for detecting an inclination of the middle joint of the finger with respect to a direction of gravity, The estimation unit is Calculating an angle of an MP joint of the finger based on an angle of the PIP joint of the finger and an inclination of the middle joint of the finger with respect to the direction of gravity detected by the first posture sensor; The information processing device according to claim 1 .

3. A second posture sensor is further provided for detecting an inclination of the back of the hand with respect to the direction of gravity, The estimation unit is Calculating a relative angle of the proximal phalanges of the fingers with respect to the back of the hand as an angle of the MP joint based on an inclination of the back of the hand with respect to the direction of gravity detected by the second posture sensor; The information processing device according to claim 2 .

4. The second posture sensor is provided on the back of the hand, on the base bones of the index finger and middle finger, in an area between the MP joint and the wrist. The information processing device according to claim 3 .

5. Further comprising a third attitude sensor for detecting an inclination of the wrist with respect to the direction of gravity; The estimation unit is calculating a tilt of the wrist with respect to the back of the hand with respect to the direction of gravity as a tilt angle of the wrist based on a tilt of the wrist with respect to the direction of gravity detected by the third posture sensor and a tilt of the back of the hand with respect to the direction of gravity detected by the second posture sensor; The information processing device according to claim 3 .

6. the first posture sensor detects a tilt of the adduction or abduction of the finger with respect to the direction of gravity; The second posture sensor detects a tilt of the palm of the hand with respect to the direction of gravity when the palm of the hand is inwardly or outwardly moved, The estimation unit is calculating information regarding a relative angle of the adduction or abduction of the fingers with respect to the dorsum of the hand based on an inclination of the adduction or abduction of the fingers with respect to the direction of gravity detected by the first posture sensor and an inclination of the adduction or abduction of the palm with respect to the direction of gravity detected by the second posture sensor; The information processing device according to claim 3 .

7. The first attitude sensor is a nine-axis sensor. The information processing device according to claim 2 .

8. The second attitude sensor is a nine-axis sensor. The information processing device according to claim 3 .

9. the third attitude sensor is a nine-axis sensor; The information processing device according to claim 5 .

10. A remote center of motion (RCM) mechanism with a double parallelogram structure using parallel links; A rotation mechanism that rotates the RCM mechanism in accordance with the rotation of the adduction or abduction of the fingers; a holding member that is fixed to a first vertex diagonally opposite to a connection point of two parallelogram structures of the RCM mechanism and holds a finger; a first link having one end fixed to a second vertex diagonally opposite to the connecting point; a second link having one end connected to the other end of the first link by a first ball joint; a third link having one end connected to the other end of the second link by a second ball joint; Equipped with the first link, the second link, and the third link form a four-joint link structure; Further comprising an exoskeleton robot, The information processing device according to claim 1 .

11. The exoskeleton robot further comprises a motor; The other end of the third link is fixed to a rotating shaft of the motor. The information processing device according to claim 10.

12. a rotation axis of the rotation mechanism and a rotation axis of the RCM mechanism are arranged in this order from the side closer to the wrist of a wearer of the exoskeleton robot, that is, the rotation axis of the rotation mechanism and the rotation axis of the RCM mechanism; The information processing device according to claim 10.

13. The rotation axis of the rotation mechanism is located at the MP joint of the finger of the wearer of the exoskeleton robot, The rotation axis of the RCM mechanism is located within a predetermined range from the MP joint of the finger of the wearer of the exoskeleton robot in a direction from the MP joint of the finger to the PIP joint. The information processing device according to claim 10.

14. The estimation unit estimates a posture of the hand and fingers based on a state of the exoskeleton robot and an angle of the joint detected by the detection unit. The information processing device according to claim 10.

15. Using a non-contact sensor, a distance from the non-contact sensor to the finger, which changes depending on an angle of at least a part of the joint of the finger, is detected, and an angle of the joint is detected in a non-contact manner based on the detected distance; The posture of the hand and fingers is estimated based on the detected angles of the joints. An information processing method comprising: The non-contact sensor includes a light-emitting element that emits light and a light-receiving element that detects reflected light of the light, The light emitting element irradiates light onto the joint of the finger, The light receiving element detects the light reflected by the joint of the finger, Detecting a distance to the finger based on the intensity of the reflected light detected by the light receiving element; The non-contact sensor is provided on a middle joint of at least one of an index finger, a middle finger, a ring finger, and a little finger, The light emitting device irradiates light to at least one of a distal phalanx and a proximal phalanx of the finger, The light receiving element detects the light reflected by at least one of a distal phalanx and a proximal phalanx of the finger, In the non-contact detection of the angle of the joint, a relative angle of the distal phalange of the finger with respect to the middle phalange of the finger is calculated as an angle of the DIP joint of the finger based on a distance between the distal phalange of the finger and the non-contact sensor calculated based on the intensity of the reflected light; In estimating the posture of the fingers, a relative angle of the proximal phalange of the finger with respect to the middle phalange of the finger is calculated as an angle of the PIP joint of the finger, and the posture of the fingers is estimated based on the angle of the DIP joint and the angle of the PIP joint of the finger. Information processing methods.

16. A detection process for detecting a distance from a non-contact sensor to a finger, the distance varying depending on an angle of at least a part of a joint of the finger, and detecting an angle of the joint in a non-contact manner based on the detected distance; an estimation process of estimating a posture of the hand and fingers based on the angles of the joints detected in the detection process; An information processing program for causing a computer to execute the following: The non-contact sensor includes a light-emitting element that emits light and a light-receiving element that detects reflected light of the light, The light emitting element irradiates light onto the joint of the finger, The light receiving element detects the light reflected by the joint of the finger, The detection process includes: Detecting a distance to the finger based on the intensity of the reflected light detected by the light receiving element; The non-contact sensor is provided on a middle joint of at least one of an index finger, a middle finger, a ring finger, and a little finger, The light emitting device irradiates light to at least one of a distal phalanx and a proximal phalanx of the finger, The light receiving element detects the light reflected by at least one of a distal phalanx and a proximal phalanx of the finger, The detection process includes: calculating a relative angle of the distal phalange of the finger with respect to the middle phalange of the finger as an angle of a DIP joint of the finger based on a distance between the distal phalange of the finger and the non-contact sensor calculated based on the intensity of the reflected light; The estimation process includes: A relative angle of the proximal phalanx of the finger with respect to the middle phalanx of the finger is calculated as an angle of the PIP joint of the finger, and a posture of the finger is estimated based on the angle of the DIP joint of the finger and the angle of the PIP joint of the finger. Information processing program.

Images