Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) is provided with an estimation unit (132) that estimates time-sequential information about a finger attitude on the basis of image information including an object and finger movements with respect to the object, the movements including a finger contact movement with the object. The estimation unit (132) estimates the time-series information about the finger attitude by using a first machine learning model obtained through learning so as to estimate the time-series information about the finger attitude on the basis of the image information including the object and the finger movements.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is known a technique for recording and replaying hand and finger movements in order to convey the skilled hand movements of musical instrument players, traditional craftsmen, chefs, etc. to others (such as apprentices) and to help others improve their skills. For example, a technique has been proposed that specifies a probability map showing the probability of the presence of a point of interest on the fingers in multiple projection directions based on images of the fingers projected in multiple projection directions, and estimates the three-dimensional position of the point of interest on the fingers based on the specified multiple probability maps. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 083910 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional techniques do not necessarily estimate the posture of the fingers appropriately. For example, the above-mentioned conventional techniques merely estimate the three-dimensional position of the attention point of the fingers, and do not necessarily estimate the posture of the fingers appropriately.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program capable of appropriately estimating the posture of fingers. [Means for solving the problem]

[0006] In order to solve the above problems, an information processing device of one embodiment according to the present disclosure includes an estimation unit that estimates time series information regarding the posture of the fingers based on image information including the object and the movement of the fingers relative to the object, including a contact movement of the fingers relative to the object. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating an example of information processing according to the first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of an information processing system according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 4] 11 is a diagram for explaining an example of operation of the information processing system according to the embodiment. FIG. [Diagram 5] 4A and 4B are diagrams illustrating an example of the arrangement of cameras and lighting according to the embodiment. [Figure 6] 11A and 11B are diagrams illustrating an example of a set of camera arrangements and captured images according to the embodiment. [Figure 7] 11 is a diagram showing an example of two-dimensional positions of feature points of a hand included in a captured image according to the embodiment. FIG. [Figure 8] 11 is a diagram showing an example of two-dimensional positions of feature points of a hand included in a captured image according to the embodiment. FIG. [Figure 9] 11 is a diagram showing an example of two-dimensional positions of feature points of a hand included in a captured image according to the embodiment. FIG. [Figure 10] 11 is a diagram showing an example of presentation of information regarding the posture of fingers according to the embodiment; FIG. [Figure 11] 11 is a diagram showing an example of presentation of information regarding the posture of fingers according to the embodiment; FIG. [Figure 12] 11 is a diagram for explaining an example of operation of an information processing system according to a modified example of the embodiment. FIG. [Figure 13] FIG. 13 is a diagram for explaining the finger-crossing technique in piano playing. [Figure 14]FIG. 11 is a diagram illustrating an example of the configuration of an information processing system according to a second embodiment of the present disclosure. [Figure 15] FIG. 2 is a diagram illustrating an example of a configuration of a sensor information processing device according to the embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 17] 11 is a diagram for explaining an example of operation of the information processing system according to the embodiment. FIG. [Figure 18] 11 is a diagram showing an example of how the IMU sensor according to the embodiment is attached; FIG. [Figure 19] 11 is a diagram showing an example of how the IMU sensor according to the embodiment is attached; FIG. [Figure 20] FIG. 11 is a diagram illustrating an example of the configuration of an information processing system according to a third embodiment of the present disclosure. [Figure 21] FIG. 2 is a diagram illustrating an example of a configuration of a sensor information processing device according to the embodiment. [Figure 22] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 23] 11 is a diagram for explaining an example of operation of the information processing system according to the embodiment. FIG. [Figure 24] 11 is a diagram for explaining an overview of sensing by the wearable camera according to the embodiment. FIG. [Diagram 25] 2 is a diagram for explaining the structure of the wearable camera according to the embodiment. FIG. [Figure 26] 11 is a diagram for explaining an example of operation of an information processing system according to a modified example of the embodiment. FIG. [Figure 27] FIG. 11 is a diagram illustrating an example of the configuration of an information processing system according to a fourth embodiment of the present disclosure. [Figure 28] FIG. 2 is a diagram illustrating an example of a configuration of an information processing device according to the embodiment. [Figure 29] 11 is a diagram for explaining an example of operation of the information processing system according to the embodiment. FIG. [Diagram 30] 11A to 11C are diagrams for explaining a touch action of a finger on an object according to the embodiment. [Diagram 31]11A to 11C are diagrams for explaining the finger joint angle estimation process according to the embodiment. [Diagram 32] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0009] The present disclosure will be described in the following order. 0. Introduction 1. First embodiment 1.1. Overview of the information processing system 1.2. Example of information processing system configuration 1.3. Example of information processing device configuration 1.4. Example of operation of information processing system 1.5. Camera and lighting layout examples 1.6.Examples of camera arrangements and captured images 1.7. 2D Positions of Hand Feature Points 1.8.Example of hand posture information presentation 1.9. Variations 2. Second embodiment 2.1. Finger-crossing technique for piano playing 2.2. Example of information processing system configuration 2.3. Example of configuration of sensor information processing device 2.4. Example of information processing device configuration 2.5. Example of operation of information processing system 2.6.Example of IMU sensor installation 3. Third embodiment 3.1. Example of information processing system configuration 3.2. Example of the configuration of the sensor information processing device 3.3. Example of information processing device configuration 3.4. Example of operation of information processing system 3.5. Overview of Sensing Using Wearable Cameras 3.6. Structure of wearable camera 3.7. Variations 4. Fourth embodiment 4.1. Example of information processing system configuration 4.2. Example of operation of information processing system 4.3. Example of information processing device configuration 4.4. Touching an object with fingers 4.5. Finger joint angle estimation process 5.Effects 6. Hardware Configuration

[0010] [0. Introduction] Recording and playing back the exquisite hand movements of musicians, traditional craftsmen, chefs, and other skilled artisans is extremely important for transferring the skills of experts to others (such as apprentices). In addition, recording high-speed hand movements and showing them to users is extremely effective in transferring intuitive tacit knowledge when supporting the development of skills.

[0011] However, high spatial and temporal resolution is required to record fast and precise finger movements. Conventional methods have focused on gesture recognition, but they are not always able to recognize finger movements with high accuracy.

[0012] Therefore, the information processing system according to the embodiment of the present disclosure narrows the imaging range to the range of hand movement, installs multiple high-speed cameras on a flat surface in the environment, estimates the two-dimensional position of each feature point of the hand from the image captured by the high-speed cameras, and estimates the posture of the fingers based on the two-dimensional positions of the estimated feature points. This allows the information processing system to estimate the posture of the fingers without attaching sensors or markers to the joints of the fingers. In other words, the information processing system can estimate the posture of the fingers without interfering with the movement of the fingers by attaching sensors, markers, etc. Therefore, the information processing system can appropriately estimate the posture of the fingers.

[0013] [1. First embodiment] [1.1. Overview of information processing system] From here, an overview of information processing according to the first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the first embodiment of the present disclosure.

[0014] In the example shown in Fig. 1, three high-speed cameras C1 to C3 are installed on both sides of and above the piano keyboard, and each of the three high-speed cameras C1 to C3 captures the hands of a performer playing the piano from its respective camera position. For example, each of the three high-speed cameras C1 to C3 captures the keystroke motion of fingers on the keyboard or the movement motion of moving the position of fingers relative to the keyboard.

[0015] The sensor information processing device 10 acquires three moving images captured from the respective positions of the three high-speed cameras C1 to C3. Upon acquiring the three moving images, the sensor information processing device 10 transmits the acquired three moving images to the information processing device 100.

[0016] The information processing device 100 estimates time-series information regarding the posture of fingers based on image information including the object and the motion of the fingers relative to the object, including the contact motion of the fingers relative to the object. In Fig. 1, the object is a keyboard, and the motion of the fingers relative to the object is the motion of the fingers striking the keyboard or the motion of moving the position of the fingers relative to the keyboard.

[0017] Specifically, the estimation unit 132 of the information processing device 100 estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in each video image of each camera (hereinafter also referred to as a sensor image) for each camera. For example, the estimation unit 132 of the information processing device 100 estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the video image of each camera using a machine learning model M1 that has been trained in advance to estimate the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the video image of each camera from each camera.

[0018] Next, the estimation unit 132 of the information processing device 100 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on the estimated two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the video images of each camera. Next, the estimation unit 132 of the information processing device 100 estimates time-series information of the posture of the fingers based on the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist. More specifically, the estimation unit 132 of the information processing device 100 estimates time-series information of the position, speed, acceleration, or trajectory of each joint of the fingers or each feature point of the palm, back of the hand, or wrist included in the video images of each camera, or the angle, angular velocity, or angular acceleration of each joint of the fingers (hereinafter also referred to as three-dimensional feature amount) as the time-series information of the posture of the fingers.

[0019] Next, the estimation unit 132 of the information processing device 100 stores time-series information of the estimated three-dimensional feature amount of the fingers in the three-dimensional feature amount database 123 of the storage unit 120. Furthermore, the information processing device 100 transmits the time-series information of the three-dimensional feature amount to the application server 200 with reference to the three-dimensional feature amount database 123.

[0020] The application server 200 acquires time-series information of the three-dimensional feature amount. Based on the acquired time-series information of the three-dimensional feature amount, the application server 200 generates an image that makes the time-series information of the three-dimensional feature amount visible. The application server 200 may generate content that makes it possible to output the time-series information of the three-dimensional feature amount together with sound. The application server 200 distributes the generated content to the terminal device 300 of the user.

[0021] The terminal device 300 displays an image that makes the time-series information of the three-dimensional feature amount visible. The terminal device 300 may also output the time-series information of the three-dimensional feature amount together with sound.

[0022] [1.2. Example of information processing system configuration] Next, a configuration of an information processing system according to a first embodiment of the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing a configuration example of the information processing system according to the first embodiment of the present disclosure. As shown in Fig. 2, the information processing system 1 according to the first embodiment includes a sensor information processing device 10, an information processing device 100, an application server 200, and a terminal device 300.

[0023] 2 are connected to each other so as to be able to communicate with each other via a network N (e.g., the Internet) in a wired or wireless manner. Note that the information processing system 1 shown in FIG. 2 may include any number of sensor information processing devices 10, any number of information processing devices 100, any number of application servers 200, and any number of terminal devices 300.

[0024] The sensor information processing device 10 acquires images captured by a high-speed monochrome camera or a high-speed infrared camera from the high-speed monochrome camera or the high-speed infrared camera. The sensor information processing device 10 acquires images including the object and the finger movement with respect to the object, including the touching movement of the finger with respect to the object. In addition, when the sensor information processing device 10 acquires images from the camera, it transmits image information including the object and the finger movement with respect to the object, including the touching movement of the finger with respect to the object, to the information processing device 100.

[0025] The information processing device 100 acquires image information including the finger movement on the object including the touch action of the finger on the object and the object from the sensor information processing device 10. Next, the information processing device 100 estimates time series information on the posture of the finger based on the image information including the finger movement on the object including the touch action of the finger on the object and the object. In addition, the information processing device 100 transmits the estimated time series information on the posture of the finger to the application server 200. Note that the sensor information processing device 10 and the information processing device 100 may be an integrated device. In that case, the information processing device 100 acquires an image captured by a high-speed monochrome camera or a high-speed infrared camera from the high-speed monochrome camera or the high-speed infrared camera. The information processing device 100 acquires an image including the finger movement on the object including the touch action of the finger on the object and the object.

[0026] The application server 200 acquires time-series information on the posture of the fingers estimated by the information processing device 100 from the information processing device 100. Upon acquiring the time-series information on the posture of the fingers, the application server 200 generates content (e.g., video images and audio) for presenting the time-series information on the posture of the fingers to a user. Upon generating the content, the application server 200 distributes the generated content to the terminal device 300.

[0027] The terminal device 300 is an information processing device used by a user. The terminal device 300 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a mobile phone, a PDA (Personal Digital Assistant), or the like. The terminal device 300 has a screen such as a liquid crystal display having a touch panel function, and accepts various operations on content such as images displayed on the screen, such as tapping, sliding, scrolling, etc., performed by a user's finger or a stylus. The terminal device 300 also has a speaker and outputs sound.

[0028] The terminal device 300 receives content from the application server 200. Upon receiving the content, the terminal device 300 displays the received content (e.g., a moving image) on a screen. The terminal device 300 also displays the moving image on a screen and outputs a sound (e.g., a piano sound) in accordance with the moving image.

[0029] [1.3. Example of configuration of information processing device] Next, a configuration of an information processing device according to a first embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of an information processing device according to a first embodiment of the present disclosure. As shown in Fig. 3, an information processing device 100 according to the first embodiment includes a communication unit 110, a storage unit 120, and a control unit 130.

[0030] (Communication unit 110) The communication unit 110 wirelessly communicates with external information processing devices such as the sensor information processing device 10, the application server 200, and the terminal device 300 via the network N. The communication unit 110 is realized by, for example, a network interface card (NIC) or an antenna. The network N may be a public communication network such as the Internet or a telephone line network, or may be a communication network provided within a limited area such as a local area network (LAN) or a wide area network (WAN). The network N may be wired. In that case, the communication unit 110 communicates with the external information processing device via wire.

[0031] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Various programs, setting data, etc. are stored in the storage unit 120. The storage unit 120 also has a sensor database 121, a model database 122, and a three-dimensional feature database 123, as shown in FIG.

[0032] (Sensor Database 121) The sensor database 121 stores image information acquired from the sensor information processing device 10. Specifically, the sensor database 121 stores information about images including the object and finger movements on the object, including contact movements of the finger on the object.

[0033] (Model Database 122) The model database 122 stores information about the machine learning model. Specifically, the model database 122 stores information about a first machine learning model that is trained to estimate time-series information about the posture of fingers (time-series information about three-dimensional feature amounts of fingers) based on image information including the motion of fingers and an object. For example, the model database 122 stores model data MDT1 of the first machine learning model.

[0034] The model data MDT1 may include an input layer to which image information including finger movements and an object is input, an output layer, a first element belonging to a layer other than the output layer that is any layer between the input layer and the output layer, and a second element whose value is calculated based on the first element and a weight of the first element, and the information processing device 100 may be caused to function so as to output, in accordance with the image information input to the input layer, time series information of the three-dimensional feature amounts of the fingers included in the image information input to the input layer from the output layer.

[0035] Here, it is assumed that the model data MDT1 is realized by a regression model indicated by "y=a1*x1+a2*x2+···+ai*xi". In this case, the first element included in the model data MDT1 corresponds to input data (xi) such as x1, x2, etc. Furthermore, the weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be regarded as a simple perceptron having an input layer and an output layer. When each model is regarded as a simple perceptron, the first element corresponds to any one of the nodes in the input layer, and the second element can be regarded as a node in the output layer.

[0036] Also, assume that the model data MDT1 is realized by a neural network having one or more intermediate layers, such as a deep neural network (DNN). In this case, the first element included in the model data MDT1 corresponds to any node in the input layer or intermediate layer. The second element corresponds to a next-stage node, which is a node to which a value is transmitted from the node corresponding to the first element. The weight of the first element corresponds to a connection coefficient, which is a weight considered for a value transmitted from the node corresponding to the first element to the node corresponding to the second element.

[0037] The information processing device 100 calculates time-series information of the three-dimensional feature amount of the fingers included in the image information by using a model having an arbitrary structure such as the above-mentioned regression model or neural network. Specifically, the coefficients of the model data MDT1 are set so that, when image information including the motion of the fingers and an object is input, the time-series information of the three-dimensional feature amount of the fingers included in the image information is output. The information processing device 100 calculates time-series information of the three-dimensional feature amount of the fingers by using such model data MDT1.

[0038] (3D feature database 123) The three-dimensional feature database 123 stores time-series information on three-dimensional features, which are the position, velocity, acceleration, or trajectory of each joint of the fingers or feature points of each fingertip, palm, back of the hand, or wrist contained in the video images of each camera, or the angle, angular velocity, or angular acceleration of each joint of the fingers.

[0039] (Control unit 130) The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] As shown in Fig. 3, the control unit 130 has an acquisition unit 131, an estimation unit 132, and a provision unit 133, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 3, and may be other configurations as long as they perform the information processing described below.

[0041] (Acquisition part 131) The acquisition unit 131 acquires image information including the object and a finger action on the object, including a touch action of the finger on the object. Specifically, the acquisition unit 131 acquires image information from the sensor information processing device 10. More specifically, the acquisition unit 131 acquires a plurality of pieces of image information acquired by a plurality of cameras installed so as to capture the object from a plurality of different directions. For example, the acquisition unit 131 acquires a plurality of pieces of image information captured by three or more cameras installed on both sides of the object and above the object.

[0042] (Estimation part 132) The estimation unit 132 estimates time-series information on the posture of fingers based on image information including the object and the motion of the fingers relative to the object, including the contact motion of the fingers with respect to the object. Specifically, the estimation unit 132 estimates time-series information on three-dimensional feature amounts of the fingers as the time-series information on the posture of the fingers. For example, the estimation unit 132 estimates time-series information on the position, speed, acceleration, or trajectory of each joint of the fingers or feature points of each fingertip, palm, back of the hand, or wrist, or time-series information on the angle, angular velocity, or angular acceleration of each joint of the fingers as the time-series information on the posture of the fingers.

[0043] More specifically, for each video image captured by each camera, the estimation unit 132 estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in the video image captured by each camera. For example, the estimation unit 132 estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in the video image captured by each camera using a machine learning model that has been trained in advance to estimate the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in the video image captured by each camera.

[0044] Next, the estimation unit 132 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on the estimated two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the video of each camera. Next, the estimation unit 132 estimates time-series information of the posture of the fingers based on the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist. More specifically, the estimation unit 132 estimates time-series information of the position, speed, acceleration, or trajectory of each joint of the fingers or each feature point of the palm, back of the hand, or wrist included in the video of each camera, or the angle, angular velocity, or angular acceleration of each joint of the fingers (hereinafter also referred to as three-dimensional feature amount) as the time-series information of the posture of the fingers.

[0045] The estimation unit 132 may estimate time-series information regarding the posture of fingers using a first machine learning model that has been trained to estimate time-series information regarding the posture of fingers based on image information including the motion of fingers and an object. For example, the estimation unit 132 inputs image information including the motion of fingers and an object into the first machine learning model, and estimates time-series information regarding the position, velocity, acceleration, or trajectory of each joint of fingers or feature points of each fingertip, palm, back of the hand, or wrist included in the video images of each camera, or the angle, angular velocity, or angular acceleration of each joint of fingers (hereinafter also referred to as three-dimensional feature amount) as time-series information of the posture of fingers.

[0046] (Provider 133) The providing unit 133 provides the user with time-series information on the posture of the fingers estimated by the estimating unit 132. Specifically, when the providing unit 133 acquires the time-series information on the posture of the fingers with reference to the three-dimensional feature database 123, the providing unit 133 generates content (e.g., moving images and audio) for presenting the time-series information on the posture of the fingers to the user. For example, the providing unit 133 generates an image in which the posture of the fingers and the positions, speeds, and accelerations of the feature points are represented by arrows and colors. The providing unit 133 also generates content for presenting the generated image and audio together. The providing unit 133 then distributes the generated content to the terminal device 300.

[0047] The providing unit 133 may transmit the time-series information regarding the posture of the hand and fingers to the application server 200 and provide the time-series information regarding the posture of the hand and fingers to the user via the application server 200.

[0048] [1.4. Example of operation of information processing system] Next, the operation of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG. 4. FIG. 4 is a diagram for describing an example of the operation of the information processing system according to the first embodiment of the present disclosure. In the example shown in FIG. 4, the information processing device 100 acquires sensor images 1, 2, 3, ... captured by a plurality of high-speed cameras installed in the environment. Next, the information processing device 100 inputs the acquired sensor images 1, 2, 3, ... to the machine learning model M1. The information processing device 100 estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in each of the sensor images 1, 2, 3, ... as output information of the machine learning model M1.

[0049] Next, the information processing device 100 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in each estimated sensor image and the camera parameters. Next, the information processing device 100 estimates time-series information of the three-dimensional feature amounts of the fingers based on the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist. Next, the information processing device 100 stores the time-series information of the three-dimensional feature amounts of the fingers in a database.

[0050] [1.5. Camera and lighting arrangement examples] Next, the arrangement of cameras and lighting according to the first embodiment of the present disclosure will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the arrangement of cameras and lighting according to the first embodiment of the present disclosure. In FIG. 5, a plurality of cameras are installed so as to capture an image of a keyboard, which is an object, from a plurality of different directions. Specifically, three cameras C1 to C3 are installed on both sides of and above the keyboard. The image information is a plurality of pieces of image information acquired by each of the plurality of cameras installed so as to capture an image of the object from a plurality of different directions. Specifically, the image information is a plurality of pieces of image information captured by each of three or more cameras installed on both sides of and above the object.

[0051] When taking pictures using a high-speed camera, since the amount of light is often insufficient in a general environment, infrared or visible light line or surface light sources are installed to surround the working space. In the example shown in FIG. 5, the camera lighting is installed on a gate-shaped structure surrounding the piano keys. Also, three cameras C1 to C3 are attached to a gate-shaped structure surrounding the piano keys, and each of the image information taken by each camera is taken in a state where the fingers are illuminated by a light source installed near each of the three cameras C1 to C3. In this way, the multiple cameras are attached to a gate-shaped structure surrounding the object, and each of the multiple image information is multiple image information taken in a state where the fingers are illuminated by a light source installed near each camera. This allows light to shine on the hand from the side, so that the fingers are not hidden by the hand's shadow. Note that a ring light may be attached to each camera. Also, a canopy may be provided on the performer's side to prevent the lighting from entering the eyes.

[0052] In addition, when capturing high-speed actions such as piano playing, the shutter speed needs to be increased, and it is desirable to use a monochrome camera or an infrared camera in order to ensure the amount of light so as not to affect the performer. In FIG. 5, cameras C1 to C3, which are high-speed monochrome cameras (for example, 90 fps or more), are attached to the environment. The image information captured by the cameras C1 to C3 is image information captured by a high-speed monochrome camera or a high-speed infrared camera. Note that a monochrome camera is more suitable for high-speed shooting because it also captures infrared light (increasing the amount of light with visible light affects the subject's actions). Note that an RGB camera (hereinafter also referred to as a normal camera) can also be used. The cameras are attached so that they are on one plane within a frame or room. This allows epipolar geometry to be used for calculations, and is expected to improve the calculation accuracy.

[0053] In addition, since the thumb and little finger are often hidden by the hand while playing the piano, a camera is also placed on the opposite side to the shooting direction. This prevents the thumb and little finger from being hidden by the hand. Specifically, the camera on the opposite side is placed at an angle of about 45 degrees from parallel to the ground surface. This makes it possible to track the thumb and little finger with two or more cameras even with only three cameras, as shown in Figure 5, and reduces data loss when estimating the 3D position of the fingers.

[0054] In addition, the imaging range of the camera is narrowed down to an area where the hands can be captured. Because the resolution of a camera is limited, narrowing the imaging range improves the resolution and accuracy of position estimation (for example, when capturing an image of a 1m range with a 2000px sensor, the resolution is 0.5mm). In the example shown in Fig. 5, the imaging range of cameras C1 to C3 is the range from the fingertips of the fingers of the performer's left hand H1 and right hand H2 to the wrist. In addition, the image information is image information captured with the imaging range being the range from the fingertips of the fingers to the wrist.

[0055] [1.6. Examples of camera arrangements and captured images] Next, a set of camera arrangements and captured images according to the first embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of a set of camera arrangements and captured images according to the first embodiment of the present disclosure.

[0056] In the example shown in Fig. 6, four cameras (1) to (4) are installed to capture images of the target keyboard from multiple different directions. Specifically, the four cameras (1) to (4) are installed on both sides of and above the keyboard.

[0057] Moreover, the image information is a plurality of pieces of image information acquired by a plurality of cameras installed so as to capture an image of an object from a plurality of different directions. Specifically, the image captured by camera (1) is an image captured by camera (1) installed to the left of the keyboard. Furthermore, the image captured by camera (2) is an image captured by camera (2) installed above and to the left of the keyboard. Furthermore, the image captured by camera (3) is an image captured by camera (3) installed above and to the right of the keyboard. Furthermore, the image captured by camera (4) is an image captured by camera (4) installed above and to the right of the keyboard.

[0058] [1.7. 2D position of hand feature points] Next, the two-dimensional positions of feature points of the hand included in the images captured by the cameras according to the first embodiment of the present disclosure will be described with reference to FIGS.

[0059] First, the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure. Fig. 7 shows an example of the two-dimensional positions of the feature points of the hand included in an image captured by a camera installed above the keyboard.

[0060] Next, the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure. Fig. 8 shows an example of the two-dimensional positions of the feature points of the hand included in an image captured by a camera installed to the left of the keyboard.

[0061] Next, the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the two-dimensional positions of the feature points of the hand included in the captured image according to the first embodiment of the present disclosure. Fig. 9 shows an example of the two-dimensional positions of the feature points of the hand included in an image captured by a camera installed to the right side of the keyboard.

[0062] [1.8. Examples of hand and finger posture information presentation] Next, the presentation of information regarding the posture of the fingers according to the first embodiment of the present disclosure will be described with reference to Figs. 10 to 11. First, the presentation of information regarding the posture of the fingers according to the first embodiment of the present disclosure will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of presentation of information regarding the posture of the fingers according to the first embodiment of the present disclosure. In the example shown in Fig. 10, the providing unit 133 provides an image in which the trajectory of the movement of the fingers is represented by superimposing a line. The terminal device 300 displays an image in which the trajectory of the movement of the fingers is represented by superimposing a line. In addition, the terminal device 300 outputs the sound of a piano performance together with the movement of the fingers.

[0063] Next, presentation of information regarding the posture of fingers according to the first embodiment of the present disclosure will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of presentation of information regarding the posture of fingers according to the first embodiment of the present disclosure. In the example shown in Fig. 11, the providing unit 133 provides content in which the time change of the speed, angle, etc. of the fingers is expressed in a graph. The terminal device 300 displays the content in which the time change of the speed, angle, etc. of the fingers is expressed in a graph.

[0064] [1.9. Modifications] Next, the operation of the information processing system according to the modified example of the first embodiment of the present disclosure will be described with reference to Fig. 12. Fig. 12 is a diagram for describing an example of the operation of the information processing system according to the modified example of the first embodiment of the present disclosure. The movement of the fingers is also expressed as the movement of the tendons on the back of the hand. Therefore, in the example shown in Fig. 12, the estimation unit 132 estimates time-series information regarding the posture of the fingers based on image information of the back of the hand performing the finger movement.

[0065] Specifically, the estimation unit 132 estimates time series information regarding the posture of the fingers using a second machine learning model trained to estimate time series information regarding the posture of the fingers based on image information of the back of the hand of a hand performing a finger movement. For example, the estimation unit 132 extracts image information of a characteristic region of the back of the hand from image information captured by a high-speed camera installed in the environment. For example, the estimation unit 132 extracts image information of a tendon part on the back of the hand as image information of the characteristic region of the back of the hand. Next, the estimation unit 132 estimates time series information regarding the angle of the joints of the fingers using a second machine learning model trained to estimate time series information regarding the angle of the joints of the fingers based on image information of the characteristic region of the back of the hand.

[0066] For example, the estimation unit 132 acquires image information captured by a high-speed camera installed in the environment from the sensor information processing device 10. Then, the estimation unit 132 extracts a characteristic region of the back of the hand from the acquired image information. Then, the estimation unit 132 inputs the extracted image information of the characteristic region of the back of the hand to a second machine learning model, and estimates time-series information regarding the angles of the finger joints included in the image captured by the high-speed camera.

[0067] [2. Second embodiment] [2.1. Finger-crossing technique in piano playing] Next, the finger-crossing technique in piano playing will be described with reference to Fig. 13. Fig. 13 is a diagram for explaining the finger-crossing technique in piano playing. In piano playing, there is a technique called "finger-crossing" in which the index finger crosses the thumb, and the thumb may be hidden. The dotted line in Fig. 13 indicates the position of the thumb that is hidden by the palm when the hand performing the finger-crossing technique in piano playing is viewed from directly above.

[0068] When the thumb is in the position shown by the dotted line due to "finger passing" as shown in Fig. 13, it is difficult to capture the image from any angle with a camera installed in the environment. Therefore, the information processing system 2 complements the estimation of the posture of the fingers, which are difficult to capture with a camera installed in the environment, by using sensing data detected by multiple IMU sensors installed on the user's thumb and the back of the hand.

[0069] [2.2. Example of information processing system configuration] Next, a configuration of an information processing system according to a second embodiment of the present disclosure will be described with reference to FIG. 14. FIG. 14 is a diagram showing a configuration example of an information processing system according to a second embodiment of the present disclosure. As shown in FIG. 14, the information processing system 2 according to the second embodiment is different from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 20. In addition, the information processing system 2 according to the second embodiment is different from the information processing system 1 according to the first embodiment in that it includes an information processing device 100A instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 20 will be mainly described, and detailed description of other configurations included in the information processing system 2 according to the second embodiment will be omitted.

[0070] 14 are connected to each other so as to be able to communicate with each other via a network N (e.g., the Internet) in a wired or wireless manner. Note that the information processing system 2 shown in FIG 14 may include any number of sensor information processing devices 10, any number of sensor information processing devices 20, any number of information processing devices 100A, any number of application servers 200, and any number of terminal devices 300.

[0071] The sensor information processing device 20 acquires sensing data detected by each of the IMU sensors installed on the thumb and back of the hand of the user from each of the IMU sensors. The sensor information processing device 20 also estimates the relative orientation between the IMU sensors based on the sensing data acquired from each of the IMU sensors. After estimating the relative orientation between the IMU sensors, the sensor information processing device 20 transmits information related to the estimated relative orientation between the IMU sensors to the information processing device 100A.

[0072] The information processing device 100A acquires sensing data detected by each of the multiple IMU sensors from the sensor information processing device 20. The information processing device 100A estimates the posture of the fingers that are difficult to capture by a camera installed in the environment based on the sensing data. Note that the sensor information processing device 20 and the information processing device 100A may be an integrated device. In this case, the information processing device 100A acquires sensing data detected by each of the multiple IMU sensors installed on the thumb and the back of the hand from each of the multiple IMU sensors. In addition, the information processing device 100A estimates the relative posture between the multiple IMU sensors based on the sensing data acquired from each of the multiple IMU sensors.

[0073] [2.3. Example of the configuration of a sensor information processing device] Next, a configuration of a sensor information processing device according to a second embodiment of the present disclosure will be described with reference to Fig. 15. Fig. 15 is a diagram showing a configuration example of a sensor information processing device according to a second embodiment of the present disclosure. In the example shown in Fig. 15, the sensor information processing device 20 includes a posture estimation unit and a communication unit.

[0074] Each attitude estimation unit acquires sensing data from each of the three IMU sensors 1 to 3. The attitude estimation unit estimates the relative attitude between the three IMU sensors 1 to 3 based on the sensing data acquired from each of the three IMU sensors 1 to 3. After estimating the relative attitude between the three IMU sensors 1 to 3, the attitude estimation unit outputs information about the estimated attitude to the communication unit.

[0075] The communication unit communicates with the information processing device 100A via the network N. The communication unit may wirelessly communicate with the information processing device 100A using communication standards such as Wi-Fi (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), Bluetooth Low Energy (registered trademark), ANT (registered trademark), ANT+ (registered trademark), or EnOcean Alliance (registered trademark).

[0076] The communication unit acquires information about the relative attitudes between the three IMU sensors 1 to 3 from the attitude estimation unit. When the communication unit acquires the information about the relative attitudes between the three IMU sensors 1 to 3, the communication unit transmits the acquired information about the relative attitudes to the information processing device 100A.

[0077] [2.4. Example of configuration of information processing device] Next, a configuration of an information processing device according to a second embodiment of the present disclosure will be described with reference to FIG. 16. FIG. 16 is a diagram showing a configuration example of an information processing device according to a second embodiment of the present disclosure. As shown in FIG. 16, the information processing device 100A according to the second embodiment is different from the information processing device 100 according to the first embodiment in that the information processing device 100A includes an estimation unit 132A and a sensor database 121A instead of the estimation unit 132 and the sensor database 121. Therefore, in the following description, the estimation unit 132A and the sensor database 121A will be mainly described, and detailed description of other configurations included in the information processing device 100A according to the second embodiment will be omitted.

[0078] (Sensor database 121A) The sensor database 121A differs from the sensor database 121 of the information processing device 100 according to the first embodiment in that it stores information regarding the relative orientation between multiple IMU sensors acquired from the sensor information processing device 20. The sensor database 121A stores information regarding the relative orientation between multiple IMU sensors installed on the thumb and back of the hand of the user acquired by the acquisition unit 131.

[0079] (Estimation unit 132A) The estimation unit 132A estimates time-series information on the posture of the user's fingers based on sensing data detected by multiple IMU sensors installed on the user's thumb and back of the hand. Specifically, the estimation unit 132A refers to the sensor database 121A to obtain information on the relative posture between the multiple IMU sensors installed on the user's thumb and back of the hand. The estimation unit 132A also obtains information on a model of the fingers on which the multiple IMU sensors are installed.

[0080] Next, the estimation unit 132A estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information about the relative posture between the multiple IMU sensors, information about the finger model, and information about the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in the estimated video images of each camera.

[0081] For example, when it is determined that the video of each camera does not include a specific feature point of a finger, the estimation unit 132A estimates the three-dimensional position of the specific feature point of a finger based on information on the relative orientation between the multiple IMU sensors and information on the model of the finger. When it is determined that the video of each camera includes a specific feature point of a finger but the accuracy is low, the estimation unit 132A estimates the three-dimensional position of the specific feature point of a finger by weighting and averaging the accuracy of the three-dimensional position of the specific feature point of a finger estimated based on information on the relative orientation between the multiple IMU sensors and information on the model of the finger and the accuracy of the three-dimensional position of the specific feature point of a finger estimated based on the video of each camera.

[0082] Next, the estimation unit 132A estimates time-series information of the posture of the specific finger based on the estimated three-dimensional position of the specific finger. More specifically, the estimation unit 132A estimates time-series information of the three-dimensional feature amount of the specific finger as the time-series information of the posture of the specific finger.

[0083] Furthermore, the estimation unit 132A may increase the weight of the angle of the joint of the finger to which the IMU sensor is attached, which is estimated based on information about the IMU sensor. Furthermore, when a sensor image exists for the position of the joint of the finger to which the IMU sensor is attached, the estimation unit 132A may complement the position using information about the sensor image. This is expected to not only complement the position of the hidden finger, but also improve the accuracy of estimating the angle of the joint of the hidden finger.

[0084] [2.5. Example of operation of information processing system] Next, the operation of the information processing system according to the second embodiment of the present disclosure will be described with reference to FIG. 17. FIG. 17 is a diagram for describing an example of the operation of the information processing system according to the second embodiment of the present disclosure. In the example shown in FIG. 17, similarly to FIG. 4, the information processing device 100A acquires sensor images 1, 2, 3, ... captured by each of a plurality of high-speed cameras installed in the environment. Then, the information processing device 100A inputs the acquired sensor images 1, 2, 3, ... to the machine learning model M1. The information processing device 100A estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in each of the sensor images 1, 2, 3, ... as output information of the machine learning model M1. In addition, the information processing device 100A acquires the camera parameters of each of the plurality of high-speed cameras.

[0085] 17, the information processing device 100A acquires sensing data detected by a plurality of IMU sensors 1, 2, 3, ... attached to a specific finger and the back of the hand of a user. Next, the information processing device 100A estimates a relative orientation between the plurality of IMU sensors based on the acquired sensing data. The information processing device 100A also acquires information on a model of the finger on which the plurality of IMU sensors are attached.

[0086] Next, the information processing device 100A estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information on the relative posture between the multiple IMU sensors, information on the finger model, and information on the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in the estimated video images of each camera.

[0087] For example, the information processing device 100A estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information on the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the estimated video images of each camera, as in FIG. 4. When the information processing device 100A determines that the feature points of a specific finger (for example, a finger such as the thumb hidden by finger-tying) are not included in the video images of each camera, the information processing device 100A estimates the three-dimensional positions of the feature points of the specific finger based on information on the relative posture between the multiple IMU sensors and information on the model of the finger. When the information processing device 100A determines that the feature points of the specific finger are included in the video images of each camera but the accuracy is low, the information processing device 100A estimates the three-dimensional positions of the feature points of the specific finger by weighting and averaging the accuracy of the three-dimensional positions of the feature points of the specific finger estimated based on the information on the relative posture between the multiple IMU sensors and information on the model of the finger and the accuracy of the three-dimensional positions of the feature points of the specific finger estimated based on the video images of each camera.

[0088] Next, the information processing device 100A estimates time-series information of the posture of the fingers based on the estimated three-dimensional positions of the fingers. More specifically, the information processing device 100A estimates time-series information of the three-dimensional feature amount of the fingers as time-series information of the posture of the fingers. Next, the information processing device 100 stores the time-series information of the three-dimensional feature amount of the fingers in a database.

[0089] [2.6. Example of IMU sensor installation] Next, wearing of the IMU sensor according to the second embodiment of the present disclosure will be described with reference to Figs. 18 to 19. In Figs. 18 to 19, a wearing example in which sensing data of the thumb is acquired by the IMU sensor according to the second embodiment will be described. For example, in the case of sensing the thumb by the IMU sensor according to the second embodiment, the IMU sensor is attached to two joints of the thumb and at least one other location.

[0090] First, the second embodiment will be described with reference to Fig. 18. Fig. 18 is a diagram showing an example of how to wear an IMU sensor according to a second embodiment of the present disclosure. In the example shown in Fig. 18, a first IMU sensor (IMU1) is attached in a range from the IP joint to the distal phalanx of the thumb. For example, the first IMU sensor (IMU1) has a thin and small shape and can be attached to a predetermined position of the thumb.

[0091] In addition, a second IMU sensor (IMU2) is attached to the area from the MP joint to the proximal phalanx of the thumb. For example, the second IMU sensor (IMU2) is ring-shaped and can be worn on the thumb.

[0092] In addition, a third IMU sensor (IMU3) is attached around the lunate bone of the palm. The attachment position of the third IMU sensor (IMU3) is not limited to around the lunate bone of the palm, and may be any position that is anatomically difficult to move. For example, the third IMU sensor (IMU3) is thin and small, and can be attached to a predetermined position on the palm.

[0093] Next, the attachment of an IMU sensor according to the second embodiment of the present disclosure will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example of attachment of an IMU sensor according to the second embodiment of the present disclosure. In the example shown in Fig. 19, similar to Fig. 18, a first IMU sensor (IMU1) is attached in the range from the IP joint to the distal phalanx of the thumb. Also, a second IMU sensor (IMU2) is attached in the range from the MP joint to the proximal phalanx of the thumb.

[0094] Figure 19 differs from Figure 18 in that the third IMU sensor (IMU3) is attached to the index finger, not around the lunate bone of the palm. In Figure 19, the third IMU sensor (IMU3) is ring-shaped and can be worn on the index finger.

[0095] [3. Third embodiment] In the information processing system 2 according to the second embodiment described above, an example has been described in which sensing data detected by a plurality of IMU sensors installed on the thumb and back of the hand of a user is used to supplement estimation of the posture of fingers that are difficult to capture using a camera installed in the environment. However, when a piano performance is filmed, there are many cases in which fingers other than the thumb are hidden due to finger tying or the like.

[0096] For example, when a piano performance is filmed, the middle finger or ring finger may be hidden by other fingers when the performer moves it. Therefore, in the information processing system 3 according to the third embodiment, an example will be described in which the posture estimation of the fingers, which is difficult to capture by a camera installed in the environment, is supplemented based on image information captured by a wearable camera attached to the wrist of the user and sensing data detected by an IMU sensor mounted on the wearable camera.

[0097] [3.1. Example of information processing system configuration] Next, a configuration of an information processing system according to a third embodiment of the present disclosure will be described with reference to FIG. 20. FIG. 20 is a diagram showing a configuration example of an information processing system according to a third embodiment of the present disclosure. As shown in FIG. 20, the information processing system 3 according to the third embodiment is different from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 30. In addition, the information processing system 3 according to the third embodiment is different from the information processing system 1 according to the first embodiment in that it includes an information processing device 100B instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 30 will be mainly described, and detailed description of other configurations included in the information processing system 3 according to the third embodiment will be omitted.

[0098] The various devices shown in Fig. 20 are connected to each other so as to be able to communicate with each other via a network N (for example, the Internet) in a wired or wireless manner. Note that the information processing system 3 shown in Fig. 20 may include any number of sensor information processing devices 10, any number of sensor information processing devices 30, any number of information processing devices 100B, any number of application servers 200, and any number of terminal devices 300.

[0099] The sensor information processing device 30 acquires image information captured by the wearable camera attached to the wrist of the user from the wearable camera. The sensor information processing device 30 estimates the two-dimensional positions of the feature points of the fingers included in the image based on the image information acquired from the wearable camera. For example, the sensor information processing device 30 estimates the two-dimensional positions of the feature points of the fingers, which are the positions of the joints and fingertips of the fingers included in the image, based on the image information acquired from the wearable camera. After estimating the two-dimensional positions of the feature points of the fingers, the sensor information processing device 30 transmits information related to the estimated two-dimensional positions of the feature points of the fingers to the information processing device 100B.

[0100] Furthermore, the sensor information processing device 30 acquires sensing data detected by the IMU sensor of the wearable camera from the IMU sensor of the wearable camera. The sensor information processing device 30 estimates the attitude of the wearable camera based on the sensing data acquired from the IMU sensor. Next, the sensor information processing device 30 estimates the camera parameters of the wearable camera based on the estimated attitude of the wearable camera. After estimating the camera parameters of the wearable camera, the sensor information processing device 30 transmits information on the estimated camera parameters of the wearable camera to the information processing device 100B.

[0101] The information processing device 100B acquires information on the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera from the sensor information processing device 30. The information processing device 100B also acquires information on the camera parameters of the wearable camera from the sensor information processing device 30. The information processing device 100B estimates the posture of the fingers that is difficult to capture by a camera installed in the environment, based on the information on the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera and the information on the camera parameters of the wearable camera. Note that the sensor information processing device 30 and the information processing device 100B may be an integrated device. In that case, the information processing device 100B acquires image information captured by the wearable camera attached to the user's wrist from the wearable camera. The information processing device 100B estimates the two-dimensional positions of the feature points of the fingers included in the image, based on the image information acquired from the wearable camera. For example, information processing device 100B estimates two-dimensional positions of finger feature points, which are the positions of finger joints and fingertips included in an image, based on image information acquired from a wearable camera. Information processing device 100B also acquires sensing data detected by an IMU sensor provided in the wearable camera from the IMU sensor of the wearable camera. Information processing device 100B estimates the posture of the wearable camera based on the sensing data acquired from the IMU sensor. Next, information processing device 100B estimates camera parameters of the wearable camera based on the estimated posture of the wearable camera.

[0102] [3.2. Example of the configuration of a sensor information processing device] Next, a configuration of a sensor information processing device according to a third embodiment of the present disclosure will be described with reference to Fig. 21. Fig. 21 is a diagram showing a configuration example of a sensor information processing device according to a third embodiment of the present disclosure. In the example shown in Fig. 21, the sensor information processing device 30 includes a posture estimation unit, an image processing unit, and a communication unit.

[0103] The posture estimation unit acquires sensing data detected by the IMU sensor of the wearable camera from the IMU sensor of the wearable camera. The posture estimation unit estimates the posture of the wearable camera based on the sensing data acquired from the IMU sensor. Next, the posture estimation unit estimates camera parameters of the wearable camera based on the estimated posture of the wearable camera. After estimating the camera parameters of the wearable camera, the posture estimation unit outputs information related to the estimated camera parameters of the wearable camera to the communication unit.

[0104] The image processing unit acquires image information captured by the wearable camera attached to the wrist of the user from the wearable camera. For example, the image processing unit may acquire image information captured by a depth sensor from the wearable camera. The image processing unit estimates two-dimensional positions of feature points of fingers included in the image based on the image information acquired from the wearable camera. For example, the image processing unit estimates two-dimensional positions of feature points of fingers included in the image using a machine learning model trained to estimate two-dimensional positions of feature points of fingers included in the image based on image information acquired from the wearable camera. When the image processing unit estimates the two-dimensional positions of the feature points of fingers, it outputs information on the estimated two-dimensional positions of the feature points of fingers to the communication unit.

[0105] The communication unit communicates with the information processing device 100B via the network N. The communication unit may wirelessly communicate with the information processing device 100B using communication standards such as Wi-Fi (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), Bluetooth Low Energy (registered trademark), ANT (registered trademark), ANT+ (registered trademark), or EnOcean Alliance (registered trademark).

[0106] The communication unit acquires information on the camera parameters of the wearable camera from the posture estimation unit. The communication unit also acquires information on the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera from the image processing unit. When the communication unit acquires the information on the camera parameters and the information on the two-dimensional positions of the feature points of the fingers, it transmits the acquired information on the camera parameters and the information on the two-dimensional positions of the feature points of the fingers to information processing device 100B.

[0107] [3.3. Example of configuration of information processing device] Next, a configuration of an information processing device according to a third embodiment of the present disclosure will be described with reference to FIG. 22. FIG. 22 is a diagram showing a configuration example of an information processing device according to a third embodiment of the present disclosure. As shown in FIG. 22, the information processing device 100B according to the third embodiment is different from the information processing device 100 according to the first embodiment in that it includes an estimation unit 132B and a sensor database 121B instead of the estimation unit 132 and the sensor database 121. Therefore, in the following description, the estimation unit 132B and the sensor database 121B will be mainly described, and detailed description of other configurations included in the information processing device 100B according to the third embodiment will be omitted.

[0108] (Sensor database 121B) Sensor database 121B differs from sensor database 121 of information processing device 100 according to the first embodiment in that it stores information on the camera parameters of the wearable camera acquired from sensor information processing device 30 and information on the two-dimensional positions of feature points of fingers included in an image captured by the wearable camera. Sensor database 121A stores information on the camera parameters acquired by acquisition unit 131 and information on the two-dimensional positions of feature points of fingers.

[0109] (Estimation part 132B) The estimation unit 132B estimates time-series information on the posture of the user's fingers based on image information captured by a wearable camera attached to the user's wrist. For example, the estimation unit 132B estimates information on the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera using a machine learning model trained to estimate the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera based on the image information captured by the wearable camera.

[0110] The wearable camera further includes an IMU sensor, and estimation unit 132B estimates time-series information about the posture of the fingers based on sensing data detected by the IMU sensor. Specifically, estimation unit 132B refers to sensor database 121B to acquire information about the camera parameters of the wearable camera and information about the two-dimensional positions of feature points of the fingers included in the image captured by the wearable camera.

[0111] Note that estimation unit 132B may acquire sensing data detected by an IMU sensor of the wearable camera from the wearable camera, and estimate the attitude of the wearable camera based on the sensing data detected by the IMU sensor. Next, estimation unit 132B may estimate camera parameters of the wearable camera based on the estimated attitude of the wearable camera.

[0112] Estimation unit 132B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information about the camera parameters of the wearable camera, information about the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera, and information about the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the estimated video images of each camera.

[0113] For example, the estimation unit 132B calculates the three-dimensional positions and certainty of the feature points of the fingers for each combination of cameras based on images captured in stereo by any two cameras among a plurality of high-speed cameras and a wearable camera installed in the environment. Next, when the estimation unit 132B determines that the feature points of the predetermined fingers are not included in the video images of the cameras, it estimates the three-dimensional positions of the feature points of the predetermined fingers (positions of the finger joints and the fingertip positions) by weighting the three-dimensional positions of the feature points of the predetermined fingers (positions of the finger joints and the fingertip positions) for each combination by the calculated certainty and averaging them.

[0114] Next, the estimation unit 132B estimates time-series information of the posture of the specific finger based on the estimated three-dimensional position of the specific finger. More specifically, the estimation unit 132B estimates time-series information of the three-dimensional feature amount of the specific finger as the time-series information of the posture of the specific finger.

[0115] [3.4. Example of operation of information processing system] Next, the operation of the information processing system according to the third embodiment of the present disclosure will be described with reference to FIG. 23. FIG. 23 is a diagram for describing an example of the operation of the information processing system according to the third embodiment of the present disclosure. In the example shown in FIG. 23, similarly to FIG. 4, the information processing device 100B acquires sensor images 1, 2, 3, ... captured by each of a plurality of high-speed cameras installed in the environment. Then, the information processing device 100B inputs the acquired sensor images 1, 2, 3, ... to the machine learning model M1. The information processing device 100B estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in each of the sensor images 1, 2, 3, ... as output information of the machine learning model M1. In addition, the information processing device 100B acquires the camera parameters of each of the plurality of high-speed cameras.

[0116] 23, the information processing device 100B acquires image information captured by a wearable camera attached to the wrist of a user. Next, the information processing device 100B estimates information related to the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera, using a machine learning model trained to estimate the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera based on the image information captured by the wearable camera.

[0117] Furthermore, information processing device 100B acquires sensing data detected by the IMU sensor of the wearable camera from the wearable camera. Then, information processing device 100B estimates the attitude of the wearable camera (IMU sensor) based on the acquired sensing data. Then, information processing device 100B estimates the camera parameters of the wearable camera based on the estimated attitude of the wearable camera (IMU sensor).

[0118] Next, information processing device 100B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information about the camera parameters of the wearable camera, information about the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera, and information about the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the estimated video images of each camera.

[0119] For example, the information processing device 100B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information on the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the estimated video images of each camera, as in FIG. 4. In addition, the information processing device 100B calculates the three-dimensional positions of the feature points of the fingers for each combination of cameras and their accuracy based on images stereoscopically captured by any two cameras among a plurality of high-speed cameras and a wearable camera installed in the environment. Next, when the information processing device 100B determines that the video images of each camera do not include the feature points of a specific finger (for example, a middle finger or a ring finger hidden by other fingers), it estimates the three-dimensional positions of the feature points of the specific finger (positions of the finger joints and the fingertip positions) by weighting the three-dimensional positions of the feature points of the specific finger (positions of the finger joints and the fingertip positions) for each combination by the calculated accuracy and averaging them.

[0120] Next, the information processing device 100B estimates time-series information of the posture of the fingers based on the estimated three-dimensional positions of the fingers. More specifically, the information processing device 100B estimates time-series information of the three-dimensional feature amount of the fingers as time-series information of the posture of the fingers. Next, the information processing device 100B stores the time-series information of the three-dimensional feature amount of the fingers in a database.

[0121] [3.5. Overview of sensing using wearable cameras] Next, an overview of sensing by a wearable camera according to a third embodiment of the present disclosure will be described with reference to Fig. 24. Fig. 24 is a diagram for explaining an overview of sensing by a wearable camera according to a third embodiment of the present disclosure.

[0122] As shown on the left side of Fig. 24, the wearable camera HC is attached to the wrist of the user and captures the palm side of the user's hand. The wearable camera HC captures the range R1 shown on the left side of Fig. 24. For example, the range R1 indicates a range that spreads in a cone shape from the camera position of the wearable camera HC toward the palm side of the user's hand.

[0123] When the wearable camera HC captures an image of the range R1, an image G1 as shown in the center of Fig. 24 is obtained. For example, image G1 includes the DIP joints and fingertips of the user's fingers close to the fingertips. Sensor information processing device 30 extracts the positions of the finger joints and fingertips included in the image as feature points of the fingers, based on the image information acquired from wearable camera HC.

[0124] The wearable camera HC captures the palm side of the user's hand using a normal camera or a depth sensor. An infrared light source may be attached to the vicinity of the camera of the wearable camera HC. The camera may be replaced with a Time-of-Flight (TOF) sensor. The posture of the wearable camera HC itself is estimated from sensing data from an IMU sensor attached in the same place as the camera.

[0125] In this way, the wearable camera HC can capture the palm side of the hand, thereby complementing information about the fingers that could not be captured by a camera attached to the environment. Also, capturing images of the palm side with the wearable camera HC makes it possible to track the fingertips without them being hidden by other fingers.

[0126] [3.6. Structure of wearable camera] Next, the structure of the wearable camera according to the third embodiment of the present disclosure will be described with reference to Fig. 25. Fig. 25 is a diagram for describing the structure of the wearable camera according to the third embodiment of the present disclosure. As shown in Fig. 25, the wearable camera HC includes a camera C4 that is a normal camera or a depth sensor. Since the wearable camera HC is attached to the wrist to capture an image of the palm of the hand, the position of the camera C4 needs to protrude from the band.

[0127] The wearable camera HC also includes an IMU sensor (IMU 4). The IMU sensor (IMU 4) is attached inside the body of the wearable camera HC.

[0128] The wearable camera HC also includes a band B1 for fastening the camera to the wrist.

[0129] The wearable camera HC may also be provided with a marker MR1 around the band for tracking from an external sensor.

[0130] [3.7. Modifications] Next, an operation of an information processing system according to a modified example of the third embodiment of the present disclosure will be described with reference to Fig. 26. Fig. 26 is a diagram for describing an example of an operation of an information processing system according to a modified example of the third embodiment of the present disclosure. Fig. 26 describes an example in which the information processing system 3 estimates time-series information on the posture of fingers based on image information from a wearable camera and image information from a high-speed camera installed in the environment, without using sensing data from an IMU sensor of the wearable camera.

[0131] In the example shown in FIG. 26, similarly to FIG. 23, the information processing device 100B acquires sensor images 1, 2, 3, ... captured by multiple high-speed cameras installed in the environment. Next, the information processing device 100B inputs the acquired sensor images 1, 2, 3, ... to the machine learning model M1. The information processing device 100B estimates the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist contained in each of the sensor images 1, 2, 3, ... as output information of the machine learning model M1. In addition, the information processing device 100B acquires camera parameters of each of the multiple high-speed cameras.

[0132] Furthermore, information processing device 100B estimates the attitude of the wearable camera based on acquired sensor images 1, 2, 3, .... Next, information processing device 100B estimates camera parameters of the wearable camera based on the estimated attitude of the wearable camera.

[0133] Next, information processing device 100B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information about the camera parameters of the wearable camera, information about the two-dimensional positions of the feature points of the fingers included in the image captured by the wearable camera, and information about the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the estimated video images of each camera.

[0134] [4. Fourth embodiment] In the information processing system 4 according to the fourth embodiment, a contact sensor for detecting contact with an object is mounted inside the object. Then, the information processing device 100C of the information processing system 4 according to the fourth embodiment estimates time-series information on the posture of fingers touching the object, based on sensing data related to the contact of the fingers with the object.

[0135] [4.1. Example of information processing system configuration] Next, a configuration of an information processing system according to a fourth embodiment of the present disclosure will be described with reference to FIG. 27. FIG. 27 is a diagram showing a configuration example of an information processing system according to a fourth embodiment of the present disclosure. As shown in FIG. 27, the information processing system 4 according to the fourth embodiment is different from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 40. In addition, the information processing system 4 according to the fourth embodiment is different from the information processing system 1 according to the first embodiment in that it includes an information processing device 100C instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 40 will be mainly described, and detailed description of other configurations included in the information processing system 4 according to the fourth embodiment will be omitted.

[0136] The sensor information processing device 40 acquires sensing data related to the contact of the finger with the object from a contact sensor mounted inside the object. When the sensor information processing device 40 acquires the sensing data related to the contact of the finger with the object, the sensor information processing device 40 transmits the sensing data to the information processing device 100C.

[0137] The information processing device 100C acquires sensing data related to the contact of fingers with an object from the sensor information processing device 40. The information processing device 100C estimates time-series information on the posture of the fingers that contact the object based on the sensing data. Note that the sensor information processing device 40 and the information processing device 100C may be an integrated device. In that case, the information processing device 100C acquires sensing data related to the contact of fingers with an object from a contact sensor mounted inside the object.

[0138] [4.2. Example of operation of information processing system] Next, an operation of the information processing system according to the fourth embodiment of the present disclosure will be described with reference to Fig. 28. Fig. 28 is a diagram for describing an example of the operation of the information processing system according to the fourth embodiment of the present disclosure. In the example shown in Fig. 28, similar to the information processing devices according to the first to third embodiments, the information processing device 100C estimates the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist based on information on the two-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist included in the video images of each camera.

[0139] The information processing device 100C also acquires contact information of the fingers on the object from the sensor information processing device 40. Then, the information processing device 100C estimates the fingers that have touched the object based on the three-dimensional positions of the feature points of the finger joints, palm, back of the hand, and wrist and the contact information of the fingers on the object. The information processing device 100C also acquires a model of the fingers to identify the fingers that have touched the object. Then, the information processing device 100C estimates the posture of the fingers that have touched the object based on the estimated fingers that have touched the object and the acquired model of the fingers.

[0140] [4.3. Example of information processing device configuration] Next, a configuration of an information processing device according to a fourth embodiment of the present disclosure will be described with reference to FIG. 29. FIG. 29 is a diagram showing a configuration example of an information processing device according to a fourth embodiment of the present disclosure. As shown in FIG. 29, the information processing device 100C according to the fourth embodiment is different from the information processing device 100 according to the first embodiment in that it includes an estimation unit 132C and a sensor database 121C instead of the estimation unit 132 and the sensor database 121. Therefore, in the following description, the estimation unit 132C and the sensor database 121C will be mainly described, and detailed description of other configurations included in the information processing device 100C according to the fourth embodiment will be omitted.

[0141] (Sensor Database 121C) The sensor database 121C differs from the sensor database 121 of the information processing device 100 according to the first embodiment in that it stores sensing data related to contact of fingers with an object acquired from the sensor information processing device 40. The sensor database 121C stores sensing data related to contact of fingers with an object acquired by the acquisition unit 131.

[0142] (Estimation unit 132C) The estimation unit 132C estimates time-series information on the posture of the fingers that have contacted the object based on sensing data detected by a contact sensor that detects the contact action of the fingers on the object. Specifically, the estimation unit 132C acquires contact information of the fingers on the object from the sensor information processing device 40. Next, the estimation unit 132C estimates the fingers that have contacted the object based on the three-dimensional positions of the feature points of the joints of the fingers, the palm, the back of the hand, and the wrist, and the contact information of the fingers on the object. In addition, the estimation unit 132C acquires a model of the fingers to identify the fingers that have contacted the object. Next, the estimation unit 132C estimates information on the posture of the fingers that have contacted the object based on the estimated fingers that have contacted the object and the acquired model of the fingers. For example, the estimation unit 132C estimates the joint angle of the fingers that have contacted the object as information on the posture of the fingers that have contacted the object. The finger joint angle estimation process performed by the estimation unit 132C will be described in detail with reference to FIG.

[0143] [4.4. Touching of fingers against an object] Next, a touch action of a finger on an object according to the fourth embodiment of the present disclosure will be described with reference to FIG. 30. FIG. 30 is a diagram for describing a touch action of a finger on an object according to the fourth embodiment of the present disclosure. In the example shown in FIG. 30, the object O2 is, for example, a piano keyboard. In addition, a contact sensor FS for detecting contact with the object is mounted inside the object O2. In FIG. 30, when the index finger of the player's hand H1 and the object O2 come into contact with each other at a point P1 on the upper surface of the object O2, the contact sensor FS detects the contact of the index finger with the object O2. When the contact sensor FS detects the contact of the index finger with the object O2, it transmits contact information between the object O2 and the index finger to the sensor information processing device 40.

[0144] [4.5. Finger joint angle estimation process] Next, a finger joint angle estimation process according to the fourth embodiment of the present disclosure will be described with reference to FIG. 31. FIG. 31 is a diagram for describing a finger joint angle estimation process according to the fourth embodiment of the present disclosure. In the example shown in FIG. 31, a user's finger presses a point P1 on the upper surface of an object O3. For example, when a user's finger presses a point P1 located at one end of an object O3, which is a keyboard, the end of the keyboard close to the pressing position P1 moves downward, and the end of the keyboard far from the pressing position P1 moves upward, thereby changing the position of the object O3. In FIG. 31, the position of the object O3 before the finger touches the object O3 is indicated by a dotted line. In addition, the position of the object O3 in a state in which the finger touches the object O3 is indicated by a solid line.

[0145] The estimation unit 132 estimates time series information on the posture of the fingers that contact the object based on the position information of the object before the contact action of the fingers on the object, the amount of change in the position of the object before and after the contact action of the fingers on the object, and the contact position information of the fingers on the object. In Fig. 31, the estimation unit 132 estimates time series information on the posture of the fingers that contact the object based on the position information of the object before the contact action of the fingers on the object (position information of the dotted line), the amount of change in the position of the object before and after the contact action of the fingers on the object O3 (amount of change in the position between the dotted line and the solid line), and information on the contact position P1 of the fingers on the object O3.

[0146] More specifically, the estimation unit 132 estimates the angle of the PIP joint of the finger based on the distance between the MP joint and the PIP joint of the finger, the distance between the PIP joint and the fingertip of the finger, the position of the MP joint of the finger, and the position of the fingertip of the finger as time-series information on the posture of the finger in contact with the object. In Fig. 31, the estimation unit 132 estimates the angle θ of the PIP joint of the finger based on the distance L1 between the position P3 of the MP joint of the finger and the position P2 of the PIP joint, the distance L2 between the position P2 of the PIP joint of the finger and the position P1 of the fingertip, the position P3 of the MP joint of the finger, and the position P1 of the fingertip of the finger. For example, the estimation unit 132 estimates the position P3 of the MP joint of the finger, the position P2 of the PIP joint, and the position P1 of the fingertip, which are included in the image information, based on the image information of a high-speed camera installed in the environment. Next, the estimation unit 132 calculates the distance L1 between the position P3 of the MP joint of the finger and the position P2 of the PIP joint, and the distance L2 between the position P2 of the PIP joint of the finger and the position P1 of the fingertip. Next, the estimation unit 132 estimates the angle θ of the PIP joint of the finger using the cosine theorem based on the calculated distances L1 and L2 and the estimated position P3 of the MP joint and the position P1 of the fingertip. Note that the DIP joint of the finger moves in synchronization with the PIP joint of the finger, so it is omitted in the calculation.

[0147] [5. Effects] As described above, the information processing device 100 according to the embodiment or the modified example thereof of the present disclosure includes the estimation unit 132. The estimation unit 132 estimates time-series information regarding the posture of fingers based on image information including the object and a motion of the fingers relative to the object, including a touch motion of the fingers relative to the object. The estimation unit 132 also estimates time-series information regarding the posture of fingers using a first machine learning model trained to estimate time-series information regarding the posture of fingers based on image information including the motion of the fingers and the object.

[0148] This allows the information processing device 100 to estimate the posture of the fingers without attaching sensors or markers to the joints of the fingers. That is, the information processing device 100 can estimate the posture of the fingers without hindering the movement of the fingers by attaching sensors, markers, etc. Therefore, the information processing device 100 can appropriately estimate the posture of the fingers during the movement of the fingers relative to an object, including the contact movement of the fingers relative to the object, such as the fingers playing the piano.

[0149] In addition, the estimation unit 132 estimates time series information regarding the posture of the fingers, such as the position, velocity, acceleration, or trajectory of each joint of the fingers or characteristic points of each fingertip, palm, back of the hand, or wrist, or time series information regarding the angle, angular velocity, or angular acceleration of each joint of the fingers.

[0150] This allows the information processing device 100 to appropriately estimate not only the three-dimensional positions of the fingers but also the angles of the finger joints, thereby making it possible to more appropriately estimate the posture of the fingers.

[0151] The image information is image information captured by a high-speed monochrome camera or a high-speed infrared camera.

[0152] As a result, even when the shutter speed is increased to capture high-speed finger movements, the information processing device 100 can ensure a sufficient amount of light without dazzling the user performing the finger movements, thereby making it possible to appropriately estimate the posture of the fingers.

[0153] The image information is a plurality of pieces of image information acquired by a plurality of cameras that are installed so as to capture an image of an object from a plurality of different directions.

[0154] As a result, when the information processing device 100 captures an image from one direction, even if the fingers are hidden by other fingers, they can be covered by capturing an image from another direction, thereby enabling the information processing device 100 to more appropriately estimate the posture of the fingers.

[0155] In addition, the multiple cameras are attached to a gate-like structure surrounding the object, and each of the multiple image information is captured while the fingers are illuminated by a light source installed near each camera.

[0156] As a result, the information processing device 100 can capture images of high-speed movements of fingers while ensuring a sufficient amount of light, and can therefore more appropriately estimate the posture of the fingers.

[0157] The image information is a plurality of pieces of image information captured by three or more cameras installed on both sides of the object and above the object.

[0158] As a result, when the information processing device 100 captures an image from one direction, even if the fingers are hidden by other fingers, they can be covered by capturing an image from another direction, thereby enabling the information processing device 100 to more appropriately estimate the posture of the fingers.

[0159] The image information is image information captured within a range from the fingertips to the wrist.

[0160] In this way, the information processing device 100 can improve the resolution and accuracy of estimating the posture of the hand and fingers by narrowing the imaging range, and can therefore more appropriately estimate the posture of the hand and fingers.

[0161] The estimation unit 132 also estimates time-series information about the posture of the fingers based on image information of the back of the hand performing the finger movement. The estimation unit 132 also estimates time-series information about the posture of the fingers using a second machine learning model trained to estimate time-series information about the posture of the fingers based on image information of the back of the hand performing the finger movement.

[0162] This allows the information processing device 100 to more appropriately estimate the posture of the fingers based on an image of the back of the hand, which is easier to capture compared to images of fingers moving at high speed.

[0163] Furthermore, the estimation unit 132 estimates time-series information regarding the posture of the user's fingers based on sensing data detected by a plurality of IMU sensors attached to the user's thumb and the back of the hand.

[0164] This enables the information processing device 100 to complement the estimation of the posture of fingers hidden by other fingers or the like.

[0165] Furthermore, the estimation unit 132 estimates time-series information regarding the posture of the user's fingers based on image information captured by a wearable camera attached to the user's wrist.

[0166] This enables the information processing device 100 to complement the estimation of the posture of fingers hidden by other fingers or the like.

[0167] The wearable camera further includes an IMU sensor, and estimation unit 132 estimates time-series information regarding the posture of the fingers based on sensing data detected by the IMU sensor.

[0168] This enables the information processing device 100 to more accurately complement the estimation of the posture of fingers hidden by other fingers or the like.

[0169] The estimation unit 132 estimates time-series information on the posture of the fingers in contact with the object based on sensing data detected by a contact sensor that detects the contact action of the fingers on the object. The estimation unit 132 estimates time-series information on the posture of the fingers in contact with the object based on the position information of the object before the contact action of the fingers on the object, the amount of change in the position of the object before and after the contact action of the fingers on the object, and the contact position information of the fingers on the object. The estimation unit 132 estimates the angle of the PIP joint of the finger based on the distance between the MP joint and the PIP joint of the finger, the distance between the PIP joint and the fingertip of the finger, the position of the MP joint of the finger, and the position of the fingertip of the finger as time-series information on the posture of the fingers in contact with the object.

[0170] This enables the information processing device 100 to complement the estimation of the posture of fingers hidden by other fingers or the like.

[0171] The object is a keyboard, and the motion of the fingers on the object is a keystroke motion of the fingers on the keyboard or a movement motion of moving the position of the fingers relative to the keyboard.

[0172] This enables the information processing device 100 to appropriately estimate the posture of the fingers while playing the piano.

[0173] The information processing device 100 further includes a providing unit 133. The providing unit 133 provides the time-series information on the posture of the fingers estimated by the estimating unit 132 to a user.

[0174] This enables the information processing device 100 to convey dexterous finger movements to others (such as disciples) and support the others in improving their skills.

[0175] [6. Hardware configuration] An information device such as the information processing device 100 according to the above-described embodiment or modification is realized by a computer 1000 having a configuration as shown in FIG. 29. FIG. 29 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of an information processing device such as the information processing device 100. The information processing device 100 according to the above-described embodiment or modification will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0176] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0177] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0178] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment of the present disclosure or a modified example thereof, which is an example of program data 1350.

[0179] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0180] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0181] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment or its modification, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 130, etc. Also, the information processing program according to the embodiment or its modification of the present disclosure and data in the storage unit 120 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1350 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0182] The present technology can also be configured as follows. (1) an estimation unit that estimates time-series information regarding a posture of the hand and fingers based on image information including a contact motion of the hand and fingers with respect to an object and the object; An information processing device comprising: (2) The estimation unit is estimating time series information regarding the posture of the hand and fingers using a first machine learning model trained to estimate time series information regarding the posture of the hand and fingers based on image information including the motion of the hand and the object; The information processing device according to (1). (3) The estimation unit is As time-series information on the posture of the fingers, the position, velocity, acceleration, or trajectory of each joint of the fingers, or each feature point of the fingertip, palm, back of the hand, or wrist, or time-series information on the angle, angular velocity, or angular acceleration of each joint of the fingers is estimated. The information processing device according to (1) or (2). (4) The image information is image information captured by a high-speed monochrome camera or a high-speed infrared camera. The information processing device according to any one of (1) to (3). (5) The image information is a plurality of pieces of image information acquired by a plurality of cameras installed so as to photograph the object from a plurality of different directions. The information processing device according to any one of (1) to (4). (6) The plurality of cameras are attached to a gate-like structure surrounding the object, Each of the plurality of pieces of image information is a plurality of pieces of image information captured in a state in which the fingers are illuminated by a light source installed near each of the cameras. The information processing device according to (5). (7) The image information is a plurality of image information captured by three or more cameras installed on both sides of the object and above the object. The information processing device according to any one of (1) to (6). (8) The image information is image information captured with a range from the fingertips of the fingers to the wrist as a capture range. The information processing device according to any one of (1) to (7). (9) The estimation unit is estimating time-series information regarding the posture of the fingers based on image information of the back of the hand performing the finger movement; The information processing device according to any one of (1) to (8). (10) The estimation unit is Estimating time series information regarding the posture of the fingers using a second machine learning model trained to estimate time series information regarding the posture of the fingers based on image information of the back of the hand performing the finger movement. The information processing device according to (9). (11) The estimation unit is Estimating time-series information regarding the posture of the user's fingers based on sensing data detected by a plurality of IMU sensors attached to the thumb and back of the hand of the user; The information processing device according to any one of (1) to (10). (12) The estimation unit is estimating time-series information regarding the posture of the user's fingers based on the image information captured by a wearable camera attached to the user's wrist; The information processing device according to any one of (1) to (11). (13) The wearable camera further comprises an IMU sensor; The estimation unit is estimating time-series information regarding the posture of the hand and fingers based on the sensing data detected by the IMU sensor; The information processing device according to (12). (14) The estimation unit is estimating time-series information regarding a posture of the fingers that have come into contact with the object based on sensing data detected by a contact sensor that detects a contact action of the fingers with respect to the object; The information processing device according to any one of (1) to (13). (15) The estimation unit is estimating time-series information on the posture of the fingers that have come into contact with the object based on position information of the object before the contact action of the fingers on the object, an amount of change in the position of the object before and after the contact action of the fingers on the object, and contact position information of the fingers on the object; The information processing device according to (14). (16) The estimation unit is As time-series information on the posture of the finger in contact with the object, an angle of the PIP joint of the finger is estimated based on the distance between the MP joint and the PIP joint of the finger, the distance between the PIP joint and the fingertip of the finger, the position of the MP joint of the finger, and the position of the fingertip of the finger; The information processing device according to (14) or (15). (17) the object is a keyboard, The motion of the fingers on the object is a keystroke motion of the fingers on the keyboard or a movement motion of moving the position of the fingers on the keyboard. The information processing device according to any one of (1) to (16). (18) a providing unit that provides a user with time-series information regarding the posture of the fingers estimated by the estimating unit, The information processing device according to any one of (1) to (17). (19) The computer estimating time-series information regarding a posture of the hand and fingers based on image information including the object and a motion of the hand and fingers relative to the object, the motion including a contact motion of the hand and fingers relative to the object; Information processing methods. (20) Computer, an estimation unit that estimates time-series information regarding a posture of the hand and fingers based on image information including a contact motion of the hand and fingers with respect to an object and the object; A program to function as a [Explanation of symbols]

[0183] 1. Information Processing Systems 10 Sensor information processing device 100 Information processing device 110 Communications Department 120 Storage section 121 Sensor Database 122 Model Database 123 3D feature database 130 Control section 131 Acquisition Department 132 Estimation Department 133 Provision Department 200 App Server 300 Terminal Equipment

Claims

1. an estimation unit that estimates time-series information regarding a posture of the fingers based on image information including a motion of the fingers with respect to an object, the motion including a contact motion of the fingers with respect to the object, and the object; Equipped with The estimation unit is estimating time-series information regarding a posture of the finger that has come into contact with the object based on sensing data detected by a contact sensor that detects a contact action of the finger with respect to the object; estimating time-series information on the posture of the fingers that have come into contact with the object based on position information of the object before the contact action of the fingers on the object, an amount of change in the position of the object before and after the contact action of the fingers on the object, and contact position information of the fingers on the object; Information processing device.

2. The estimation unit is Estimating time series information regarding the posture of the hand and fingers using a first machine learning model trained to estimate time series information regarding the posture of the hand and fingers based on image information including the motion of the hand and fingers and the object. The information processing device according to claim 1 .

3. The estimation unit is As time-series information on the posture of the fingers, the position, velocity, acceleration, or trajectory of each joint of the fingers, or each feature point of the fingertip, palm, back of the hand, or wrist, or time-series information on the angle, angular velocity, or angular acceleration of each joint of the fingers is estimated. The information processing device according to claim 1 .

4. The image information is image information captured by a high-speed monochrome camera or a high-speed infrared camera. The information processing device according to claim 1 .

5. The image information is a plurality of pieces of image information acquired by a plurality of cameras installed so as to photograph the object from a plurality of different directions. The information processing device according to claim 1 .

6. The plurality of cameras are attached to a gate-like structure surrounding the object, Each of the plurality of pieces of image information is a plurality of pieces of image information captured in a state in which the fingers are illuminated by a light source installed near each of the cameras. The information processing device according to claim 5 .

7. The image information is a plurality of pieces of image information captured by three or more cameras installed on both sides of the object and above the object. The information processing device according to claim 1 .

8. The image information is image information captured with a range from the fingertips of the fingers to the wrist as a capture range. The information processing device according to claim 1 .

9. The estimation unit is estimating time-series information regarding the posture of the fingers based on image information of the back of the hand performing the finger movement; The information processing device according to claim 1 .

10. The estimation unit is Estimating time series information regarding the posture of the fingers using a second machine learning model trained to estimate time series information regarding the posture of the fingers based on image information of the back of the hand performing the finger movement. The information processing device according to claim 9.

11. The estimation unit is estimating time-series information regarding the posture of the user's fingers based on sensing data detected by a plurality of IMU sensors installed on the thumb and back of the hand of the user; The information processing device according to claim 1 .

12. The estimation unit is estimating time-series information regarding the posture of the user's fingers based on the image information captured by a wearable camera attached to the user's wrist; The information processing device according to claim 1 .

13. The wearable camera further comprises an IMU sensor; The estimation unit is estimating time-series information regarding the posture of the hand and fingers based on the sensing data detected by the IMU sensor; The information processing device according to claim 12.

14. The estimation unit is As time-series information regarding the posture of the finger in contact with the object, the angle of the PIP joint of the finger is estimated based on the distance between the MP joint and the PIP joint of the finger, the distance between the PIP joint and the fingertip of the finger, the position of the MP joint of the finger, and the position of the fingertip of the finger. The information processing device according to claim 1 .

15. the object is a keyboard, The motion of the fingers on the object is a keystroke motion of the fingers on the keyboard or a movement motion of moving the position of the fingers on the keyboard. The information processing device according to claim 1 .

16. a providing unit that provides a user with time-series information regarding the posture of the fingers estimated by the estimating unit, The information processing device according to claim 1 .

17. The computer 1. An information processing method for estimating time-series information regarding a posture of fingers based on image information including a touch action of the fingers with respect to an object and the object, the method comprising: estimating time-series information regarding a posture of the finger that has come into contact with the object based on sensing data detected by a contact sensor that detects a contact action of the finger with respect to the object; estimating time-series information on the posture of the fingers that have come into contact with the object based on position information of the object before the contact action of the fingers on the object, an amount of change in the position of the object before and after the contact action of the fingers on the object, and contact position information of the fingers on the object; Information processing methods.

18. Computer, an estimation unit that estimates time-series information regarding a posture of the fingers based on image information including a motion of the fingers with respect to an object, the motion including a contact motion of the fingers with respect to the object, and the object; A program for causing the device to function as a The estimation unit is estimating time-series information regarding a posture of the finger that has come into contact with the object based on sensing data detected by a contact sensor that detects a contact action of the finger with respect to the object; estimating time-series information on the posture of the fingers that have come into contact with the object based on position information of the object before the contact action of the fingers on the object, an amount of change in the position of the object before and after the contact action of the fingers on the object, and contact position information of the fingers on the object; program.

Images