Information processing device, information processing method, and information processing program

Abstract

An information processing apparatus (100) includes an estimation unit (132) that estimates time-series information about finger posture based on image information, the image information including an object and finger movement relative to the object, the movement including finger contact movement relative to the object. The estimation unit (132) estimates the time-series information about finger posture by using a first machine learning model, the first machine learning model being learned to estimate the time-series information about finger posture based on image information including the object and finger movement.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing method, and information processing program. Background Technology

[0002] Traditionally, techniques for recording and reproducing finger movements are known to transmit the excellent fine motor skills of instrumentalists, traditional craftspeople, chefs, etc., to others (students, etc.) and to support the proficiency of others. For example, a technique has been proposed in which, based on images of fingers projected in multiple projection directions, probability maps representing the probability of the presence of points of interest with respect to the fingers in the multiple projection directions are specified, and the three-dimensional position of these points of interest with respect to the fingers is estimated based on the multiple specified probability maps.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: WO 2018 / 083910 A Summary of the Invention

[0006] The technical problem to be solved by the present invention

[0007] However, in the aforementioned conventional techniques, it is not always possible to properly estimate finger posture. For example, in the prior art, only the three-dimensional position of the finger's point of interest is estimated, without necessarily estimating the finger posture appropriately.

[0008] Therefore, this disclosure proposes an information processing device, information processing method, and information processing program capable of appropriately estimating finger posture.

[0009] Solution to the problem

[0010] To solve the above problems, an information processing device includes:

[0011] The estimation unit estimates time-series information about finger posture based on image information, which includes: an object, and finger-to-object operations including finger-to-object contact operations. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating an example of information processing according to a first embodiment of the present disclosure.

[0013] Figure 2 A diagram illustrating a configuration example of an information processing system according to this embodiment.

[0014] Figure 3 This is a diagram illustrating an example configuration of the information processing apparatus according to this embodiment.

[0015] Figure 4 This is a diagram used to describe an operational example of the information processing system according to this embodiment.

[0016] Figure 5 This is a diagram illustrating an example of the arrangement of the camera and lighting according to this embodiment.

[0017] Figure 6 This is a diagram illustrating an example of a set of camera arrangements and image capture according to this embodiment.

[0018] Figure 7 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to this embodiment.

[0019] Figure 8 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to this embodiment.

[0020] Figure 9 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to this embodiment.

[0021] Figure 10 This is a diagram illustrating an example of the presentation of information regarding finger posture according to this embodiment.

[0022] Figure 11 This is a diagram illustrating an example of the presentation of information regarding finger posture according to this embodiment.

[0023] Figure 12 This is a diagram used to describe an operational example of an information processing system according to a variation of this embodiment.

[0024] Figure 13 It is a simplified diagram used to describe the finger passing method in piano playing.

[0025] Figure 14 This is a diagram illustrating a configuration example of an information processing system according to a second embodiment of the present disclosure.

[0026] Figure 15 This is a diagram illustrating an example configuration of the sensor information processing apparatus according to this embodiment.

[0027] Figure 16 This is a diagram illustrating an example configuration of the information processing apparatus according to this embodiment.

[0028] Figure 17 This is a diagram used to describe an operational example of the information processing system according to this embodiment.

[0029] Figure 18This is a diagram illustrating an example of the installation of an IMU sensor according to this embodiment.

[0030] Figure 19 This is a diagram illustrating an example of the installation of an IMU sensor according to this embodiment.

[0031] Figure 20 This is a schematic diagram illustrating a configuration example of an information processing system according to a third embodiment of the present disclosure.

[0032] Figure 21 This is a diagram illustrating an example configuration of the sensor information processing apparatus according to this embodiment.

[0033] Figure 22 This is a diagram illustrating an example configuration of the information processing apparatus according to this embodiment.

[0034] Figure 23 This is a diagram used to describe an operational example of the information processing system according to this embodiment.

[0035] Figure 24 This is a diagram used to describe the overview of sensing via a wearable camera according to this embodiment.

[0036] Figure 25 This is a diagram used to describe the structure of a wearable camera according to this embodiment.

[0037] Figure 26 This is a diagram used to describe an operational example of an information processing system according to a variation of this embodiment.

[0038] Figure 27 This is a schematic diagram illustrating a configuration example of an information processing system according to a fourth embodiment of the present disclosure.

[0039] Figure 28 This is a diagram illustrating an example configuration of the information processing apparatus according to this embodiment.

[0040] Figure 29 This is a diagram used to describe an operational example of the information processing system according to this embodiment.

[0041] Figure 30 This is a diagram used to describe the contact operation of a finger relative to an object according to this embodiment.

[0042] Figure 31 This is a diagram used to describe the estimation process of the joint angle of the finger according to this embodiment.

[0043] Figure 32 This is a hardware configuration diagram illustrating an example of a computer that performs the functions of an information processing device. Detailed Implementation

[0044] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, in the following embodiments, the same reference numerals will be used for the same parts and repeated descriptions will be omitted.

[0045] This disclosure will be described in the following order of items.

[0046] 0. Introduction

[0047] 1. First Implementation Method

[0048] 1.1. Overview of Information Processing Systems

[0049] 1.2. Configuration Example of an Information Processing System

[0050] 1.3. Configuration Example of Information Processing Device

[0051] 1.4. Operational Examples of Information Processing Systems

[0052] 1.5. Example of camera and lighting arrangement

[0053] 1.6. An example of a set of camera setups and image capture.

[0054] 1.7. Two-dimensional position of feature points of the hand

[0055] 1.8. Example of presenting information about finger posture

[0056] 1.9. Variations

[0057] 2. Second Implementation Method

[0058] 2.1. Finger passing techniques in piano performance

[0059] 2.2. Configuration Example of an Information Processing System

[0060] 2.3. Configuration Example of Sensor Information Processing Device

[0061] 2.4. Configuration Example of Information Processing Device

[0062] 2.5. Operational Examples of Information Processing Systems

[0063] 2.6. IMU Sensor Installation Example

[0064] 3. Third Implementation Method

[0065] 3.1. Configuration Example of an Information Processing System

[0066] 3.2. Configuration Example of Sensor Information Processing Device

[0067] 3.3. Configuration Example of Information Processing Device

[0068] 3.4. Operational Examples of Information Processing Systems

[0069] 3.5. Overview of Wearable Camera Sensing

[0070] 3.6. Structure of Wearable Cameras

[0071] 3.7. Variations

[0072] 4. Fourth Implementation Method

[0073] 4.1. Configuration Example of an Information Processing System

[0074] 4.2. Operational Examples of Information Processing Systems

[0075] 4.3. Configuration Example of Information Processing Device

[0076] 4.4. Finger contact operation relative to an object

[0077] 4.5. Processing for estimating finger joint angles

[0078] 5. Effects

[0079] 6. Hardware Configuration

[0080] [0. Introduction]

[0081] Recording and reproducing the exquisite fine motor skills of instrumentalists, traditional craftspeople, chefs, and others is crucial for transmitting these skills to others, such as students. Furthermore, with the aid of skilled practitioners, recording high-speed finger movements and presenting them to users is highly effective for intuitively conveying tacit knowledge.

[0082] However, high-speed and detailed finger movement recording requires both high spatial and temporal resolution. Traditionally, there are many cases that emphasize gesture recognition, but they are not always able to recognize finger movements with high accuracy.

[0083] Therefore, the information processing system according to the embodiments of this disclosure narrows the shooting range to the operating range of the hand, mounts multiple high-speed cameras on a plane in the environment, estimates the two-dimensional position of each feature point of the hand from the images captured by the high-speed cameras, and estimates the finger posture based on the estimated two-dimensional position of the feature points. Therefore, the information processing system can estimate the finger posture without mounting sensors or markers on the finger joints, etc. That is, the information processing system can estimate the finger posture without hindering finger operation due to the mounting of sensors, markers, etc. Therefore, the information processing system is able to appropriately estimate the finger posture.

[0084] [1. First Implementation Method]

[0085] [1.1. Overview of Information Processing Systems]

[0086] Here, we will refer to Figure 1 A summary of information processing according to a first embodiment of this disclosure is described. Figure 1 This is a diagram illustrating an example of information processing according to a first embodiment of the present disclosure.

[0087] exist Figure 1 In the example shown, three high-speed cameras C1 to C3 are mounted on either side and above the piano keyboard, and each of the three high-speed cameras C1 to C3 captures images of the pianist's hands from its respective position. For example, each of the three high-speed cameras C1 to C3 captures the key-pressing operations of the fingers relative to the keyboard or the movement operations of the fingers relative to the position of the keyboard.

[0088] The sensor information processing unit 10 acquires each of three moving images captured from corresponding positions of three high-speed cameras C1 to C3. While acquiring the three moving images, the sensor information processing unit 10 sends the acquired three moving images to the information processing unit 100.

[0089] The information processing device 100 estimates time-series information about finger posture based on image information including: an object, and finger-to-object operations including contact operations between the finger and the object. Figure 1 In this context, the object is the keyboard, and the operation of the finger relative to the object is either a keystroke operation of the finger relative to the keyboard or a movement operation of the finger relative to the position of the keyboard.

[0090] Specifically, the estimation unit 132 of the information processing device 100 estimates the two-dimensional positions of feature points including finger joints, palms, backs of hands, and wrists in each moving image (hereinafter also referred to as sensor images) of each camera. For example, the estimation unit 132 of the information processing device 100 estimates the two-dimensional positions of feature points including finger joints, palms, backs of hands, and wrists in each moving image of each camera by using a machine learning model M1, which is learned in advance to estimate the two-dimensional positions of feature points including finger joints, palms, backs of hands, and wrists in each moving image of each camera.

[0091] Subsequently, the estimation unit 132 of the information processing apparatus 100 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on the estimated two-dimensional positions of the feature points included in the moving images of each camera. Then, the estimation unit 132 of the information processing apparatus 100 estimates the temporal sequence information of the finger posture based on the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist. More specifically, the estimation unit 132 of the information processing apparatus 100 estimates the position, velocity, acceleration, or trajectory of the feature points of each joint or fingertip, palm, back of hand, or wrist included in the moving images of each camera, or the angle, angular velocity, or angular acceleration of each joint of the finger (hereinafter also referred to as three-dimensional feature quantities), as the temporal sequence information of the finger posture.

[0092] Subsequently, the estimation unit 132 of the information processing device 100 stores the estimated time-series information of the three-dimensional feature quantities of the finger in the three-dimensional feature quantity database 123 of the storage unit 120. Furthermore, the information processing device 100, referring to the three-dimensional feature quantity database 123, sends the time-series information of the three-dimensional feature quantities to the application server 200.

[0093] Application server 200 acquires time-series information of three-dimensional feature quantities. Based on the acquired time-series information of the three-dimensional feature quantities, application server 200 generates an image that allows visual recognition of the time-series information of the three-dimensional feature quantities. Note that application server 200 can generate content in which the time-series information of the three-dimensional feature quantities can be output along with sound. Application server 200 distributes the generated content to the user's terminal device 300.

[0094] The terminal device 300 displays an image that enables visual recognition of time-series information of three-dimensional features. Furthermore, the terminal device 300 can output the time-series information of the three-dimensional features along with sound.

[0095] [1.2. Configuration Example of an Information Processing System]

[0096] Next, refer to Figure 2 This describes the configuration of an information processing system according to a first embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating a configuration example of an information processing system according to a first embodiment of the present disclosure. For example... Figure 2 As shown, the information processing system 1 according to the first embodiment includes a sensor information processing device 10, an information processing device 100, an application server 200, and a terminal device 300.

[0097] Figure 2The various devices shown are communicatively connected via a network N (e.g., the Internet) in a wired or wireless manner. It should be noted that... Figure 2 The information processing system 1 shown may include any number of sensor information processing devices 10, any number of information processing devices 100, any number of application servers 200, and any number of terminal devices 300.

[0098] The sensor information processing device 10 acquires images captured by a high-speed monochrome camera or a high-speed infrared camera. The sensor information processing device 10 also acquires images including objects and finger operations relative to the objects (including finger contact operations). Furthermore, when acquiring images from the camera, the sensor information processing device 10 transmits image information including objects and finger operations relative to the objects (including finger contact operations) to the information processing device 100.

[0099] The information processing device 100 acquires image information from the sensor information processing device 10, which includes an object and finger operations relative to the object (including finger contact operations relative to the object). Subsequently, the information processing device 100 estimates time-series information about the finger's posture based on the image information including the finger operations relative to the object (including finger contact operations relative to the object) and the object. Furthermore, the information processing device 100 transmits the estimated time-series information about the finger's posture to the application server 200. It should be noted that the sensor information processing device 10 and the information processing device 100 can be integrated. In this case, the information processing device 100 acquires images captured by a high-speed monochrome camera or a high-speed infrared camera. The information processing device 100 acquires images including an object and finger operations relative to the object (including finger contact operations relative to the object).

[0100] Application server 200 obtains time-series information about finger postures estimated by information processing device 100. While obtaining the time-series information about finger postures, application server 200 generates content (e.g., motion graphics or audio) to present the time-series information about finger postures to a user. When generating the content, application server 200 distributes the generated content to terminal device 300.

[0101] Terminal device 300 is an information processing device used by a user. Terminal device 300 is implemented as, for example, a smartphone, tablet computer, notebook computer (PC), mobile phone, personal digital assistant (PDA), etc. Furthermore, terminal device 300 includes a screen with touch panel functionality (e.g., an LCD display) and receives various operations (e.g., tapping, swiping, and scrolling) from the user on content (e.g., images displayed on the screen) using fingers, styluses, etc. In addition, terminal device 300 includes a speaker and outputs voice.

[0102] Terminal device 300 receives content from application server 200. When receiving content, terminal device 300 displays the received content (e.g., moving images) on the screen. Furthermore, terminal device 300 displays moving images on the screen and outputs sound (e.g., piano sounds) based on the moving images.

[0103] [1.3. Example of Information Processing Device Configuration]

[0104] Next, we will refer to Figure 3 The configuration of the information processing apparatus according to the first embodiment of this disclosure is described. Figure 3 This is a diagram illustrating an example configuration of an information processing apparatus according to a first embodiment of the present invention. Figure 3 As shown, the information processing apparatus 100 according to the first embodiment includes a communication unit 110, a storage unit 120, and a control unit 130.

[0105] (Communication Unit 110)

[0106] Communication unit 110 wirelessly communicates with external information processing devices (e.g., sensor information processing device 10, application server 200, or terminal device 300) via network N. Communication unit 110 is implemented using, for example, a network interface card (NIC), an antenna, etc. Network N can be a public communication network such as the Internet or a telephone network, or it can be a communication network located in a limited area such as a local area network (LAN) or a wide area network (WAN). Note that network N can be a wired network. In this case, communication unit 110 performs wired communication with the external information processing device.

[0107] (Storage Unit 120)

[0108] Storage unit 120 is implemented using, for example, semiconductor storage elements (e.g., random access memory (RAM) or flash memory) or storage devices (e.g., hard disk or optical disk). Storage unit 120 stores various programs, settings data, etc. Figure 3 As shown, the storage unit 120 includes a sensor database 121, a model database 122, and a three-dimensional feature database 123.

[0109] (Sensor Database 121)

[0110] Sensor database 121 stores image information acquired from sensor information processing device 10. Specifically, sensor database 121 stores information about operations of a finger relative to an object (including finger contact operations with the object) and images of the object.

[0111] (Model Database 122)

[0112] Model database 122 stores information about machine learning models. Specifically, model database 122 stores information about a first machine learning model that learns to estimate time-series information about finger poses based on image information including object and finger manipulation (time-series information of three-dimensional feature quantities of the fingers). For example, model database 122 stores model data MDT1 of the first machine learning model.

[0113] The model data MDT1 may include an input layer, an output layer, a first element belonging to any layer from the input layer to the output layer but not the output layer, and a second element whose value is calculated based on the first element and its weight. The model data MDT1 can be used by the information processing device 100 to output time-series information of the three-dimensional features of the finger from the output layer based on the image information input to the input layer, the finger being included in the image information input to the input layer.

[0114] Here, assume the model data MDT1 is derived from "y = a1*x1 + a2*x2 + ... + ai*xi". In this case, the first element included in the model data MDT1 corresponds to the input data (xi), for example, x1 and x2. Furthermore, the weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be viewed as a simple perceptron with an input layer and an output layer. When each model is considered a simple perceptron, the first element can be considered as any node included in the input layer, and the second element can be considered as a node included in the output layer.

[0115] Furthermore, assume that the model data MDT1 is implemented by a neural network (e.g., a deep neural network (DNN)) with one or more intermediate layers. In this case, the first element included in the model data MDT1 corresponds to any node included in the input layer or intermediate layers. Furthermore, the second element corresponds to a node in the next stage, which is the node from which values ​​are transmitted from the node corresponding to the first element to the node corresponding to the second element. Additionally, the weight of the first element corresponds to a connection coefficient, which is a weight considered for the values ​​transmitted from the node corresponding to the first element to the node corresponding to the second element.

[0116] The information processing device 100 uses a model with arbitrary structure (e.g., the regression model or neural network described above) to calculate time-series information of the three-dimensional features of the finger included in the image information. Specifically, in the model data MDT1, when image information including finger operations and objects is input, coefficients are set to output time-series information of the three-dimensional features of the finger included in the image information. The information processing device 100 uses such model data MDT1 to calculate the time-series information of the three-dimensional features of the finger.

[0117] (3D Feature Database 123)

[0118] The three-dimensional feature database 123 stores time-series information of three-dimensional features, which are the position, velocity, acceleration or trajectory of feature points of each joint of the finger or each fingertip, palm, back of hand or wrist in the motion image of each camera, or the angle, angular velocity or angular acceleration of each joint of the finger.

[0119] (Control Unit 130)

[0120] The control unit 130 is implemented by using RAM as a working area through a central processing unit (CPU), microprocessor unit (MPU), etc., to execute various programs (corresponding to examples of information processing programs) stored in the storage device within the information processing apparatus 100. Furthermore, the control unit 130 is implemented using an integrated circuit, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0121] like Figure 3 As shown, the control unit 130 includes an acquisition unit 131, an estimation unit 132, and a provisioning unit 133, and implements or performs the information processing actions described below. It should be noted that the internal configuration of the control unit 130 is not limited to... Figure 3 The configuration shown is different, but it can be another configuration, as long as the information processing described later is performed.

[0122] (Acquisition Unit 131)

[0123] The acquisition unit 131 acquires image information including finger operations relative to an object (including finger contact operations relative to the object) and the object itself. Specifically, the acquisition unit 131 acquires image information from the sensor information processing device 10. More specifically, the acquisition unit 131 acquires multiple image information captured by each of a plurality of cameras mounted to capture images of the object from multiple different directions. For example, the acquisition unit 131 acquires multiple image information captured by three or more cameras mounted on both sides of the object and above the object.

[0124] (Estimation Unit 132)

[0125] The estimation unit 132 estimates temporal series information about the finger's posture based on image information including finger operations relative to an object (including finger contact operations relative to the object) and the object itself. Specifically, the estimation unit 132 estimates temporal series information about the three-dimensional features of the finger as temporal series information about the finger's posture. For example, the estimation unit 132 estimates the position, velocity, acceleration, or trajectory of feature points of each joint of the finger or each fingertip, palm, back of the hand, or wrist, or the angle, angular velocity, or angular acceleration of each joint of the finger as temporal series information about the finger's posture.

[0126] More specifically, estimation unit 132 estimates the two-dimensional positions of feature points including finger joints, palm, back of hand, and wrist in each motion image of each camera. For example, estimation unit 132 estimates the two-dimensional positions of feature points including finger joints, palm, back of hand, and wrist in each motion image of each camera by using a machine learning model that has been learned in advance to estimate the two-dimensional positions of feature points including finger joints, palm, back of hand, and wrist in each motion image of each camera.

[0127] Subsequently, estimation unit 132 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on the estimated two-dimensional positions of the feature points included in the moving images of each camera. Then, estimation unit 132 estimates the temporal series information of the finger pose based on the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist. More specifically, estimation unit 132 estimates the temporal series information of the finger pose as the position, velocity, acceleration, or trajectory of the feature points of each finger or each joint of each fingertip, palm, back of hand, or wrist included in the moving images of each camera, or the angle, angular velocity, or angular acceleration of each joint of the finger (hereinafter also referred to as three-dimensional feature quantities).

[0128] Furthermore, the estimation unit 132 can estimate time-series information about finger posture using a first machine learning model, wherein the first machine learning model learns to estimate time-series information about finger posture based on image information including finger manipulation and objects. For example, the estimation unit 132 inputs image information including finger manipulation and objects into the first machine learning model and estimates time-series information including the position, velocity, acceleration, or trajectory of feature points of each finger or each joint of each fingertip, palm, back of hand, or wrist in moving images of each camera, or the angle, angular velocity, or angular acceleration of each joint of the finger (hereinafter also referred to as three-dimensional feature quantities), as time-series information about finger posture.

[0129] (Provide Unit 133)

[0130] The providing unit 133 provides the user with time-series information about finger posture estimated by the estimation unit 132. Specifically, when time-series information about finger posture is obtained by referring to the three-dimensional feature database 123, the providing unit 133 generates content (e.g., motion images or audio) to present the time-series information about finger posture to the user. For example, the providing unit 133 generates an image in which the finger posture and the position, velocity, and acceleration of feature points are represented by arrows or colors. Furthermore, the providing unit 133 generates content that presents the generated image and sound. Subsequently, the providing unit 133 distributes the generated content to the terminal device 300.

[0131] Note that the providing unit 133 can send time-series information about finger posture to the application server 200, and provide the user with time-series information about finger posture via the application server 200.

[0132] [1.4. Operational Example of an Information Processing System]

[0133] Next, we will refer to Figure 4 The operation of the information processing system according to the first embodiment of this disclosure is described. Figure 4 This is a diagram illustrating an operational example of an information processing system according to a first embodiment of the present invention. Figure 4 In the example shown, the information processing device 100 acquires sensor images 1, 2, 3, ... by taking pictures with multiple high-speed cameras installed in the environment. The information processing device 100 then inputs the acquired sensor images 1, 2, 3, ... into a machine learning model M1. The information processing device 100 estimates the two-dimensional locations of feature points of the finger joints, palm, back of hand, and wrist included in each of the sensor images 1, 2, 3, ... as the output information of the machine learning model M1.

[0134] Subsequently, the information processing device 100 estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on the two-dimensional positions of these feature points included in the estimated sensor images and camera parameters. Then, the information processing device 100 estimates the time-series information of the three-dimensional features of the finger based on the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist. Finally, the information processing device 100 stores the time-series information of the three-dimensional features of the finger in a database.

[0135] [1.5. Example of camera and lighting arrangement]

[0136] Next, we will refer to Figure 5The arrangement of the camera and lighting according to a first embodiment of this disclosure is described. Figure 5 This is a diagram illustrating an example of the arrangement of a camera and lighting according to a first embodiment of this disclosure. Figure 5 In this design, multiple cameras are installed to photograph the keyboard as the object from multiple different directions. Specifically, three cameras, C1 to C3, are installed on both sides and above the keyboard. Furthermore, the image information is obtained from multiple images captured by each of the multiple cameras installed to photograph the object from multiple different directions. Specifically, the image information consists of multiple images captured by three or more cameras installed on both sides and above the object.

[0137] When using a high-speed camera, the amount of light is often insufficient in normal environments, therefore, infrared or visible light sources or surface light sources are installed to surround the workspace. Figure 5 In the example shown, the camera lighting is mounted on a gate-like structure surrounding the piano keyboard. Furthermore, three cameras, C1 to C3, are attached to this gate-like structure, and each image captured by each camera is taken while the fingers are illuminated by a light source mounted near each of the three cameras. In this way, multiple cameras are attached to the gate-like structure surrounding the object, and each of the multiple image messages is a series of images taken while the fingers are illuminated by a light source mounted near each camera. As a result, the hands are also illuminated from the side, and the fingers are not hidden by shadows. Note that a ring light can be attached to each camera. Alternatively, a awning can be provided on the player's side to prevent the light from entering their eyes.

[0138] Furthermore, in situations involving high-speed actions such as photographing a piano performance, it is necessary to increase the shutter speed, and it is desirable to use a monochrome or infrared camera to ensure sufficient light so as not to affect the performer. Figure 5 In this setup, cameras C1 to C3, acting as high-speed monochrome cameras (e.g., above 90 fps), are attached to the environment. The image information captured by cameras C1 to C3 is either image information captured by a high-speed monochrome camera or a high-speed infrared camera. Note that monochrome cameras are also better suited for high-speed shooting by capturing infrared light (utilizing visible light to increase the amount of light affecting the operator's actions), and RGB cameras (also referred to as normal cameras below) can also be used. Furthermore, the cameras are mounted in a frame or room so that they are positioned on a plane. Therefore, epipolar geometry can be used for calculations, and improvements in calculation accuracy can be expected.

[0139] Furthermore, since the thumb and little finger are often hidden by the hand during piano playing, the camera is also positioned on the side opposite to the shooting direction. This covers the area where the thumb and little finger are hidden by the hand. Specifically, the camera is mounted by tilting it to the opposite side within a range from parallel to the ground surface to approximately 45 degrees. Therefore, even when only three cameras are present, as... Figure 5 As shown, the thumb and little finger can also be tracked by two or more cameras, reducing data loss during finger 3D position estimation.

[0140] Furthermore, the camera's imaging range is narrowed to the area capable of capturing a hand. Because the camera's resolution is limited, the resolution and accuracy of position estimation improve when the shooting range is narrowed (e.g., when capturing a 1m area with a 2000px sensor, the resolution is 0.5mm). Figure 5 In the example shown, the shooting range of cameras C1 to C3 is from the fingertips to the wrists of the performer's left hand H1 and right hand H2. Furthermore, the image information is captured using the range from the fingertips to the wrists as the shooting range.

[0141] [1.6. An example of a set of cameras and image capture]

[0142] Next, we will refer to Figure 6 Describes a set of camera arrangements and image capture according to a first embodiment of the present disclosure. Figure 6 This is a diagram illustrating a set of camera devices and examples of captured images according to a first embodiment of the present disclosure.

[0143] exist Figure 6 In the example shown, four cameras (1) to (4) are installed to photograph the keyboard as the object from multiple different directions. Specifically, the four cameras (1) to (4) are installed on both sides and above the keyboard.

[0144] Furthermore, the image information consists of multiple image pieces acquired by each of multiple cameras installed to capture the object from multiple different directions. Specifically, the image captured by camera (1) is the image captured by camera (1) installed on the left side of the keyboard. The image captured by camera (2) is the image captured by camera (2) installed on the upper left side of the keyboard. The image captured by camera (3) is the image captured by camera (3) installed on the upper right side of the keyboard. The image captured by camera (4) is the image captured by camera (4) installed on the right side of the keyboard.

[0145] [1.7. Two-dimensional position of feature points of the hand]

[0146] Next, we will refer to Figures 7 to 9 The two-dimensional positions of feature points of a hand included in an image captured by each camera according to a first embodiment of the present disclosure are described.

[0147] First, refer to Figure 7 The two-dimensional positions of feature points of a hand included in a captured image are described according to a first embodiment of the present invention. Figure 7 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to a first embodiment of the present disclosure; Figure 7 An example is shown showing the two-dimensional locations of feature points of a hand in an image taken by a camera mounted above the keyboard.

[0148] Next, we will refer to Figure 8 The two-dimensional positions of feature points of a hand included in a captured image are described according to a first embodiment of the present disclosure. Figure 8 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to a first embodiment of this disclosure. Figure 8 An example is shown showing the two-dimensional positions of feature points of a hand in an image taken by a camera mounted on the left side of the keyboard.

[0149] Next, we will refer to Figure 9 The two-dimensional positions of feature points of a hand included in a captured image are described according to a first embodiment of the present invention. Figure 9 This is a diagram illustrating an example of the two-dimensional positions of feature points of a hand included in a captured image according to a first embodiment of the present disclosure. Figure 9 An example is shown showing the two-dimensional locations of feature points of a hand in an image taken by a camera mounted on the right side of the keyboard.

[0150] [1.8. Example of presenting information about finger posture]

[0151] Next, we will refer to Figure 10 and Figure 11 This describes the presentation of information regarding finger posture according to a first embodiment of the present disclosure. First, reference will be made to... Figure 10 Describe it. Figure 10 This is a diagram illustrating an example of the presentation of information regarding finger posture according to a first embodiment of this disclosure. Figure 10 In the example shown, providing unit 133 provides an image in which the movement trajectory of the finger is represented by overlapping lines. Terminal device 300 displays the image, in which the movement trajectory of the finger is represented by overlapping lines. Furthermore, terminal device 300 outputs piano playing sounds along with the finger movements.

[0152] Next, we will refer to Figure 11 The presentation of information regarding finger posture according to a first embodiment of this disclosure. Figure 11 This is a diagram illustrating an example of the presentation of information regarding finger posture according to a first embodiment of this disclosure. Figure 11 In the example shown, providing unit 133 provides content in which, for example, the temporal changes in the speed and angle of a finger are graphically represented. Terminal device 300 displays content in which, for example, the temporal changes in the speed and angle of a finger are graphically represented.

[0153] [1.9. Variation]

[0154] Next, we will refer to Figure 12 The operation of an information processing system according to a variation of the first embodiment of this disclosure is described. Figure 12 This is a diagram illustrating an operational example of an information processing system according to a variation of the first embodiment of this disclosure. Finger manipulation also occurs on the back of the hand as tendon manipulation. Therefore, in Figure 12 In the embodiment shown, the estimation unit 132 estimates time-series information about the finger posture based on image information of the back of the hand performing the finger operation.

[0155] Specifically, estimation unit 132 estimates time-series information about finger posture using a second machine learning model. This second machine learning model is trained to estimate time-series information about finger posture based on image information of the back of the hand performing the finger operation. For example, estimation unit 132 extracts image information of feature regions on the back of the hand from image information captured by a high-speed camera mounted in the environment. For example, estimation unit 132 extracts image information of a portion of the tendons on the back of the hand as image information of feature regions on the back of the hand. Subsequently, estimation unit 132 uses the second machine learning model to estimate time-series information about the angles of the finger joints. This second machine learning model is trained to estimate time-series information about the angles of the finger joints based on the image information of feature regions on the back of the hand.

[0156] For example, estimation unit 132 acquires image information captured by a high-speed camera installed in the environment from sensor information processing device 10. Then, estimation unit 132 extracts feature regions of the back of the hand from the acquired image information. Subsequently, estimation unit 132 inputs the image information of the extracted feature regions of the back of the hand into a second machine learning model and estimates time-series information about the angles of the finger joints included in the image captured by the high-speed camera.

[0157] [2. Second Implementation]

[0158] [2.1. Finger passing techniques in piano performance]

[0159] Next, we will refer to Figure 13 Describe the finger techniques used in piano playing. Figure 13This is a simplified diagram used to describe the finger passing technique in piano playing. Piano playing includes a technique known as "finger passing," in which the index finger plays across the thumb, and the thumb can be concealed. Figure 13 The dotted line shown indicates the position of the thumb, which is hidden behind the palm and cannot be seen when the hand is viewed from directly above during a piano performance using the finger-passing method.

[0160] Because in Figure 13 The “finger transmission” shown in the diagram indicates that when the thumb is in the position indicated by the dotted line, it is difficult for a camera attached to the environment to take a picture from any angle. Therefore, in the information processing system 2, the posture estimation of the fingers that are difficult to take pictures with a camera attached to the environment is supplemented by sensing data detected by multiple IMU sensors mounted on the thumb and back of the user's hand.

[0161] [2.2. Configuration Example of an Information Processing System]

[0162] Next, we will refer to Figure 14 The configuration of the information processing system according to the second embodiment of this disclosure is described. Figure 14 This is a diagram illustrating a configuration example of an information processing system according to a second embodiment of the present disclosure. (See diagram for example.) Figure 14 As shown, the information processing system 2 according to the second embodiment differs from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 20. Furthermore, the information processing system 2 according to the second embodiment differs in that it includes an information processing device 100A instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 20 will be described primarily, and detailed descriptions of other configurations included in the information processing system 2 according to the second embodiment will be omitted.

[0163] exist Figure 14 The various devices shown are communicatively connected via a network N (e.g., the Internet) in a wired or wireless manner. It should be noted that... Figure 14 The information processing system 2 shown may include any number of sensor information processing devices 10, any number of sensor information processing devices 20, any number of information processing devices 100A, any number of application servers 200, and any number of terminal devices 300.

[0164] The sensor information processing device 20 acquires sensing data detected by each of the plurality of IMU sensors mounted on the thumb and back of the user's hand. Furthermore, the sensor information processing device 20 estimates the relative pose between the plurality of IMU sensors based on the sensing data acquired from each of the plurality of IMU sensors. While estimating the relative pose between the plurality of IMU sensors, the sensor information processing device 20 transmits information about the estimated relative pose between the plurality of IMU sensors to the information processing device 100A.

[0165] Information processing device 100A acquires sensing data detected by each of the plurality of IMU sensors from sensor information processing device 20. Information processing device 100A estimates the finger posture, which is difficult to capture with a camera mounted in the environment, based on the sensing data. It should be noted that sensor information processing device 20 and information processing device 100A can be integrated devices. In this case, information processing device 100A acquires sensing data detected by each of the plurality of IMU sensors mounted on the thumb and back of the user's hand. Furthermore, information processing device 100A estimates the relative posture between the plurality of IMU sensors based on the sensing data acquired from each of the plurality of IMU sensors.

[0166] [2.3. Configuration Example of Sensor Information Processing Device]

[0167] Next, we will refer to Figure 15 The configuration of the sensor information processing apparatus according to the second embodiment of the present disclosure is described. Figure 15 This is a diagram illustrating an example configuration of a sensor information processing apparatus according to a second embodiment of the present disclosure. Figure 15 In the embodiment shown, the sensor information processing device 20 includes a posture estimation unit and a communication unit.

[0168] Each pose estimation unit acquires sensing data from each of the three IMU sensors 1 to 3. Based on the sensing data acquired from each of the three IMU sensors 1 to 3, the pose estimation unit estimates the relative pose between the three IMU sensors 1 to 3. While estimating the relative pose between the three IMU sensors 1 to 3, the pose estimation unit outputs information related to the estimated pose to the communication unit.

[0169] The communication unit communicates with the information processing device 100A via network N. Furthermore, the communication unit can wirelessly communicate with the information processing device 100A using communication technologies such as Wi-Fi, ZigBee, Bluetooth, Bluetooth Low Energy, ANT, ANT+, and EnOcean Alliance.

[0170] The communication unit obtains information about the relative posture between the three IMU sensors 1 to 3 from the posture estimation unit. While obtaining information about the relative posture between the three IMU sensors 1 to 3, the communication unit transmits the obtained information about the relative posture to the information processing device 100A.

[0171] [2.4. Example of Information Processing Device Configuration]

[0172] Next, we will refer to Figure 16 The configuration of the information processing apparatus according to the second embodiment of this disclosure is described. Figure 16 This is a diagram illustrating an example configuration of an information processing apparatus according to a second embodiment of the present disclosure. (See diagram for example.) Figure 16 As shown, the information processing apparatus 100A according to the second embodiment differs from the information processing apparatus 100 according to the first embodiment in that it provides an estimation unit 132A and a sensor database 121A, instead of an estimation unit 132 and a sensor database 121. Therefore, in the following description, the estimation unit 132A and the sensor database 121A will be described primarily, and detailed descriptions of other configurations included in the information processing apparatus 100A according to the second embodiment will be omitted.

[0173] (Sensor Database 121A)

[0174] The sensor database 121A differs from the sensor database 121 of the information processing apparatus 100 according to the first embodiment in that it stores information about the relative postures between multiple IMU sensors acquired from the sensor information processing apparatus 20. Sensor database 121A stores information about the relative postures between multiple IMU sensors mounted on the thumb and back of the user's hand, acquired by the acquisition unit 131.

[0175] (Estimation Unit 132A)

[0176] The estimation unit 132A estimates time-series information about the user's finger posture based on sensing data detected by multiple IMU sensors mounted on the user's thumb and back of the hand. Specifically, the estimation unit 132A refers to the sensor database 121A to obtain information about the relative posture between the multiple IMU sensors mounted on the user's thumb and back of the hand. Furthermore, the estimation unit 132A obtains information about a model of the finger with the multiple IMU sensors mounted on it.

[0177] Subsequently, the estimation unit 132A estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on information about the relative pose between the multiple IMU sensors, information about the finger model, and estimation information about the two-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist included in the moving images of each camera.

[0178] For example, if it is determined that the feature points of a predetermined finger are not included in the moving images of each camera, the estimation unit 132A estimates the three-dimensional position of the feature points of the predetermined finger based on information about the relative pose between the multiple IMU sensors and information about the finger model. Furthermore, if the feature points of the predetermined finger are included in the moving images of each camera but the accuracy is determined to be low, the estimation unit 132A estimates the three-dimensional position of the feature points of the predetermined finger by weighting and averaging the accuracy of the three-dimensional position estimated based on the information about the relative pose between the multiple IMU sensors and the information about the finger model, and the accuracy of the three-dimensional position estimated based on the moving images of each camera.

[0179] Subsequently, the estimation unit 132A estimates the time-series information of the pose of the predetermined finger based on the estimated three-dimensional position of the predetermined finger. More specifically, the estimation unit 132A estimates the time-series information of the three-dimensional feature quantities of the predetermined finger as the time-series information of the pose of the predetermined finger.

[0180] Furthermore, the estimation unit 132A can add weights to the estimated values ​​based on information about the IMU sensor for the angles of the finger joints to which the IMU sensor is attached. Moreover, if a sensor image of the position of the finger joint attached to the IMU sensor exists, the estimation unit 132A can supplement the position using information from the sensor image. As a result, not only can the complementary position of the hidden finger be expected, but the accuracy of the angle estimation of the hidden finger joints can also be expected to improve.

[0181] [2.5. Operational Example of an Information Processing System]

[0182] Next, we will refer to Figure 17 The operation of the information processing system according to the second embodiment of this disclosure is described. Figure 17 This is a diagram illustrating an operational example of an information processing system according to a second embodiment of the present disclosure. Figure 17 In the example shown, similar to Figure 4 The information processing device 100A acquires sensor images 1, 2, 3, ... captured by multiple high-speed cameras installed in the environment. Subsequently, the information processing device 100A inputs the acquired sensor images 1, 2, 3, ... into a machine learning model M1. The information processing device 100A estimates the two-dimensional positions of feature points including finger joints, palms, backs of hands, and wrists in each of the sensor images 1, 2, 3, ... as output information of the machine learning model M1. Furthermore, the information processing device 100A acquires camera parameters for each of the multiple high-speed cameras.

[0183] In addition, Figure 17 In the information processing device 100A, sensing data is acquired from each of a plurality of IMU sensors 1, 2, 3, ... These sensors are mounted on predetermined fingers and the back of the user's hand. Subsequently, the information processing device 100A estimates the relative posture between the plurality of IMU sensors based on the acquired sensing data. Furthermore, the information processing device 100A acquires information about a model of the finger on which the plurality of IMU sensors are mounted.

[0184] Subsequently, the information processing device 100A estimates the three-dimensional position of the feature points of the finger joints, palm, back of hand, and wrist based on information about the relative posture between multiple IMU sensors, information about the finger model, and estimation information about the two-dimensional position of the feature points of the finger joints, palm, back of hand, and wrist included in the moving images of each camera.

[0185] For example, similar to Figure 4The information processing device 100A estimates the three-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist based on information about the two-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist included in the estimated moving images of each camera. Furthermore, if it is determined that the feature points of a predetermined finger are not included in the moving images of each camera (e.g., the thumb hidden by a finger clasping method), the information processing device 100A estimates the three-dimensional positions of the feature points of the predetermined finger based on information related to the relative posture between multiple IMU sensors and information related to the finger model. Moreover, if it is determined that the feature points of the predetermined finger are included in the moving images of each camera but the accuracy is low, the information processing device 100A estimates the three-dimensional positions of the feature points of the predetermined finger by weighting and averaging the accuracy of the three-dimensional positions estimated based on information about the relative posture between multiple IMU sensors and information about the finger model, and the accuracy of the three-dimensional positions estimated based on the moving images of each camera.

[0186] Subsequently, the information processing device 100A estimates the time-series information of the finger's pose based on the estimated three-dimensional position of the finger. More specifically, the information processing device 100A estimates the time-series information of the three-dimensional features of the finger as the time-series information of the finger's pose. Then, the information processing device 100A stores the time-series information of the three-dimensional features of the finger in a database.

[0187] [2.6. IMU Sensor Installation Example]

[0188] Next, we will refer to Figure 18 and Figure 19 The installation of an IMU sensor according to a second embodiment of this disclosure is described. Figure 18 and Figure 19 The following describes a wearing example where the IMU sensor acquires sensing data of the thumb according to the second embodiment. For example, when the IMU sensor according to the second embodiment senses the thumb, the IMU sensor is attached to two nodes of the thumb and at least one other location.

[0189] First, refer to Figure 18 Describe it. Figure 18 This is a diagram illustrating an example of the installation of an IMU sensor according to a second embodiment of this disclosure. Figure 18 In the example shown, a first IMU sensor (IMU1) is attached to a region extending from the IP joint of the thumb to the distal phalanx. For example, the first IMU sensor (IMU1) has a thin and small shape and is capable of being fixed to a predetermined position on the thumb.

[0190] In addition, a second IMU sensor (IMU2) is attached to a region extending from the MP joint of the thumb to the proximal phalanx. For example, the second IMU sensor (IMU2) is ring-shaped and can be fitted into the thumb.

[0191] In addition, the third IMU sensor (IMU3) is attached around the lunula of the palm. Note that the attachment location of the third IMU sensor (IMU3) is not limited to around the lunula of the palm, and can be any location as long as it is anatomically difficult to move. For example, the third IMU sensor (IMU3) has a thin and small shape and can be fixed to a predetermined location on the palm.

[0192] Next, we will refer to Figure 19 The installation of an IMU sensor according to a second embodiment of this disclosure is described. Figure 19 This is a diagram illustrating an example of the installation of an IMU sensor according to a second embodiment of this disclosure. Figure 19 In the example shown, with Figure 18 Similarly, the first IMU sensor (IMU1) is attached to the area from the IP joint of the thumb to the distal phalanx. Furthermore, the second IMU sensor (IMU2) is attached to the area from the MP joint of the thumb to the proximal phalanx.

[0193] Figure 19 and Figure 18 The difference lies in the fact that the third IMU sensor (IMU3) is attached to the index finger instead of around the crescent bone in the palm. Figure 19 In the middle, the third IMU sensor (IMU3) is ring-shaped and can be fitted onto the index finger.

[0194] [3. Third Implementation Method]

[0195] In the information processing system 2 according to the second embodiment described above, an example is described in which the posture estimation of fingers, which is difficult to perform by shooting with a camera installed in the environment, is supplemented by sensing data detected by multiple IMU sensors installed on the thumb and back of the user's hand. However, in the case of shooting a piano performance, fingers other than the thumb are often hidden due to clenching, etc.

[0196] For example, in the case of filming a piano performance, when the player moves their middle or ring finger, the middle or ring finger can be hidden by other fingers. Therefore, in the information processing system 3 according to the third embodiment, an example of estimating the posture of fingers that are difficult to capture by a camera installed in the environment will be described, based on image information captured by a wearable camera attached to the user's wrist and sensing data detected by an IMU sensor mounted on the wearable camera.

[0197] [3.1. Configuration Example of an Information Processing System]

[0198] Next, we will refer to Figure 20 The configuration of the information processing system according to the third embodiment of this disclosure is described. Figure 20 This is a schematic diagram illustrating a configuration example of an information processing system according to a third embodiment of the present disclosure. For example... Figure 20 As shown, the information processing system 3 according to the third embodiment differs from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 30. Furthermore, the information processing system 3 according to the third embodiment differs in that it includes an information processing device 100B instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 30 will be described primarily, and detailed descriptions of other configurations included in the information processing system 3 according to the third embodiment will be omitted.

[0199] Figure 20 The various devices shown can be communicatively connected via a network N (e.g., the Internet) in a wired or wireless manner. It should be noted that... Figure 20 The information processing system 3 shown may include any number of sensor information processing devices 10, any number of sensor information processing devices 30, any number of information processing devices 100B, any number of application servers 200, and any number of terminal devices 300.

[0200] The sensor information processing device 30 acquires image information captured by a wearable camera attached to the user's wrist. Based on the image information acquired from the wearable camera, the sensor information processing device 30 estimates the two-dimensional positions of feature points of a finger in the image. For example, the sensor information processing device 30 estimates the two-dimensional positions of feature points of the finger, which are the positions of finger joints or fingertips included in the image, based on the image information acquired from the wearable camera. After estimating the two-dimensional positions of the finger's feature points, the sensor information processing device 30 transmits information about the estimated two-dimensional positions of the finger's feature points to the information processing device 100B.

[0201] Furthermore, the sensor information processing unit 30 acquires sensing data detected by the IMU sensor included in the wearable camera from the IMU sensor of the wearable camera. The sensor information processing unit 30 estimates the pose of the wearable camera based on the sensing data acquired from the IMU sensor. Subsequently, the sensor information processing unit 30 estimates the camera parameters of the wearable camera based on the estimated pose. When estimating the camera parameters of the wearable camera, the sensor information processing unit 30 transmits information about the estimated camera parameters of the wearable camera to the information processing unit 100B.

[0202] Information processing device 100B acquires information from sensor information processing device 30 regarding the two-dimensional positions of feature points of a finger included in an image captured by a wearable camera. Furthermore, information processing device 100B acquires information from sensor information processing device 30 regarding camera parameters of the wearable camera. Based on the information regarding the two-dimensional positions of feature points of a finger included in an image captured by the wearable camera and the information regarding the camera parameters of the wearable camera, information processing device 100B estimates the finger posture that is difficult to capture with a camera mounted in the environment. It should be noted that sensor information processing device 30 and information processing device 100B can be integrated devices. In this case, information processing device 100B acquires image information from a wearable camera attached to the user's wrist. Information processing device 100B estimates the two-dimensional positions of feature points of a finger included in the image based on the image information acquired from the wearable camera. For example, information processing device 100B estimates the two-dimensional positions of feature points of a finger based on the image information acquired from the wearable camera, which are the positions of finger joints or fingertips included in the image. Furthermore, the information processing device 100B acquires sensing data detected by the IMU sensor included in the wearable camera from the IMU sensor of the wearable camera. The information processing device 100B estimates the pose of the wearable camera based on the sensing data acquired from the IMU sensor. Subsequently, the information processing device 100B estimates the camera parameters of the wearable camera based on the estimated pose of the wearable camera.

[0203] [3.2. Configuration Example of Sensor Information Processing Device]

[0204] Next, we will refer to Figure 21 The configuration of a sensor information processing apparatus according to a third embodiment of the present disclosure is described. Figure 21 This is a diagram illustrating an example configuration of a sensor information processing apparatus according to a third embodiment of the present disclosure. Figure 21 In the embodiment shown, the sensor information processing device 30 includes a pose estimation unit, an image processing unit, and a communication unit.

[0205] The pose estimation unit acquires sensing data detected by the IMU sensor included in the wearable camera from the IMU sensor of the wearable camera. The pose estimation unit estimates the pose of the wearable camera based on the sensing data acquired from the IMU sensor. Subsequently, the pose estimation unit estimates the camera parameters of the wearable camera based on the estimated pose. When estimating the camera parameters of the wearable camera, the pose estimation unit outputs information related to the estimated camera parameters of the wearable camera to the communication unit.

[0206] The image processing unit acquires image information captured by a wearable camera attached to the user's wrist. For example, the image processing unit may acquire image information captured by a depth sensor from the wearable camera. Based on the image information acquired from the wearable camera, the image processing unit estimates the two-dimensional positions of feature points of fingers included in the image. For example, the image processing unit estimates the two-dimensional positions of feature points of fingers included in the image by using a machine learning model that learns to estimate the two-dimensional positions of feature points of fingers included in the image based on the image information acquired from the wearable camera. After estimating the two-dimensional positions of the finger feature points, the image processing unit outputs information about the estimated two-dimensional positions of the finger feature points to the communication unit.

[0207] The communication unit communicates with the information processing device 100B via network N. Furthermore, the communication unit can wirelessly communicate with the information processing device 100B using communication methods such as Wi-Fi, ZigBee, Bluetooth, Bluetooth Low Energy, ANT, ANT+, and EnOcean Alliance.

[0208] The communication unit obtains camera parameter information about the wearable camera from the pose estimation unit. Furthermore, the communication unit obtains two-dimensional position information about feature points of the finger included in the image captured by the wearable camera from the image processing unit. When the information about the camera parameters and the information about the two-dimensional position of the finger feature points are obtained, the communication unit transmits the obtained information about the camera parameters and the obtained information about the two-dimensional position of the finger feature points to the information processing device 100B.

[0209] [3.3. Example of Information Processing Device Configuration]

[0210] Next, refer to Figure 22 The configuration of the information processing apparatus according to the third embodiment of the present disclosure is described. Figure 22 This is a diagram illustrating an example configuration of an information processing apparatus according to a third embodiment of the present disclosure. (See diagram for example.) Figure 22 As shown, the information processing apparatus 100B according to the third embodiment differs from the information processing apparatus 100 according to the first embodiment in that it provides an estimation unit 132B and a sensor database 121B instead of an estimation unit 132 and a sensor database 121. Therefore, in the following description, the estimation unit 132B and the sensor database 121B will be described primarily, and detailed descriptions of other configurations included in the information processing apparatus 100B according to the third embodiment will be omitted.

[0211] (Sensor Database 121B)

[0212] The sensor database 121B differs from the sensor database 121 of the information processing apparatus 100 according to the first embodiment in that the sensor database 121B stores information about camera parameters of the wearable camera acquired from the sensor information processing apparatus 30 and information about the two-dimensional positions of feature points of the fingers included in the image captured by the wearable camera. The sensor database 121A stores information about camera parameters acquired by the acquisition unit 131 and information about the two-dimensional positions of feature points of the fingers.

[0213] (Estimation Unit 132B)

[0214] The estimation unit 132B estimates time-series information about the user's finger posture based on image information captured by a wearable camera attached to the user's wrist. For example, the estimation unit 132B estimates information about the two-dimensional positions of feature points of the fingers included in the images captured by the wearable camera by using a machine learning model, which learns to estimate the two-dimensional positions of feature points of the fingers included in the images captured by the wearable camera based on the image information captured by the wearable camera.

[0215] Furthermore, the wearable camera also includes an IMU sensor, and the estimation unit 132B estimates time-series information about finger posture based on sensing data detected by the IMU sensor. Specifically, the estimation unit 132B refers to the sensor database 121B to obtain information about camera parameters of the wearable camera and information about the two-dimensional positions of feature points of the finger included in the image captured by the wearable camera.

[0216] Note that the estimation unit 132B can acquire sensing data detected by the IMU sensor of the wearable camera and estimate the pose of the wearable camera based on the sensing data detected by the IMU sensor. Subsequently, the estimation unit 132B can estimate the camera parameters of the wearable camera based on the estimated pose of the wearable camera.

[0217] The estimation unit 132B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on information about the camera parameters of the wearable camera, information about the two-dimensional positions of the feature points of the fingers included in the images captured by the wearable camera, and estimation information about the two-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist included in the moving images of each camera.

[0218] For example, estimation unit 132B calculates the three-dimensional position and determinism of feature points of a finger in a combination of cameras based on images viewed stereoscopically by any two of a plurality of high-speed cameras and wearable cameras installed in the environment. Subsequently, if it is determined that the feature points of a predetermined finger are not included in the motion images of each camera, estimation unit 132B estimates the three-dimensional position (position of finger joint or fingertip) of the feature points of the predetermined finger by weighting and averaging the calculated determinism with each combination of the three-dimensional position (position of finger joint or fingertip) of the feature points of the predetermined finger.

[0219] Subsequently, the estimation unit 132B estimates the time-series information of the pose of the predetermined finger based on the estimated three-dimensional position of the predetermined finger. More specifically, the estimation unit 132B estimates the time-series information of the three-dimensional feature quantities of the predetermined finger as the time-series information of the pose of the predetermined finger.

[0220] [3.4. Operational Example of an Information Processing System]

[0221] Next, refer to Figure 23 This describes the operation of an information processing system according to a third embodiment of the present disclosure. Figure 23 This is a diagram illustrating an operational example of an information processing system according to a third embodiment of the present disclosure. Figure 23 In the example shown, similar to Figure 4 The information processing device 100B acquires sensor images 1, 2, 3, ... captured by multiple high-speed cameras installed in the environment. Subsequently, the information processing device 100B inputs the acquired sensor images 1, 2, 3, ... into a machine learning model M1. The information processing device 100B estimates the two-dimensional positions of feature points of finger joints, palms, backs of hands, and wrists included in each of the sensor images 1, 2, 3, ... as the output information of the machine learning model M1. Furthermore, the information processing device 100B acquires the camera parameters of each of the multiple high-speed cameras.

[0222] In addition, Figure 23 In this process, the information processing device 100B acquires image information captured by a wearable camera attached to the user's wrist. Subsequently, the information processing device 100B estimates information about the two-dimensional positions of feature points of the fingers included in the images captured by the wearable camera by using a machine learning model. This machine learning model learns to estimate the two-dimensional positions of feature points of the fingers included in the images captured by the wearable camera based on the image information captured by the wearable camera.

[0223] Furthermore, the information processing device 100B acquires sensing data detected by the IMU sensor of the wearable camera. Subsequently, the information processing device 100B estimates the pose of the wearable camera (its IMU sensor) based on the acquired sensing data. Then, the information processing device 100B estimates the camera parameters of the wearable camera based on the estimated pose of the wearable camera (its IMU sensor).

[0224] Subsequently, the information processing device 100B estimates the three-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist based on information about the camera parameters of the wearable camera, information about the two-dimensional positions of the feature points of the fingers included in the images captured by the wearable camera, and estimation information about the two-dimensional positions of the feature points of the finger joints, palm, back of hand, and wrist included in the moving images of each camera.

[0225] For example, similar to Figure 4 The information processing device 100B estimates the three-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist based on information about the two-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist included in the estimated moving images of each camera. Furthermore, the information processing device 100B calculates the three-dimensional positions and determinism of the finger feature points in the respective camera combinations based on images viewed stereoscopically by any two of a plurality of high-speed cameras and wearable cameras installed in the environment. Subsequently, in cases where it is determined that the feature points of a predetermined finger are not included in the moving images of each camera (e.g., a middle or ring finger hidden by another finger), the information processing device 100B estimates the three-dimensional position (position of the finger joint or fingertip) of the feature points of the predetermined finger by weighted averaging with the calculated determinism in each combination.

[0226] Subsequently, the information processing device 100B estimates the time-series information of the finger's posture based on the estimated three-dimensional position of the finger. More specifically, the information processing device 100B estimates the time-series information of the three-dimensional features of the finger as the time-series information of the finger's posture. The information processing device 100B then stores the time-series information of the three-dimensional features of the finger in a database.

[0227] [3.5. Overview of Wearable Camera Sensing]

[0228] Next, we will refer to Figure 24 An overview of sensing via a wearable camera according to a third embodiment of this disclosure is described. Figure 24 This is a diagram used to describe an overview of sensing via a wearable camera according to a third embodiment of this disclosure.

[0229] like Figure 24 As shown on the left, the wearable camera HC attaches to the user's wrist and captures images of the user's palm side. The wearable camera HC captures images on... Figure 24 The left side shows an image of the R1 range. For example, the R1 range represents the range extending in a cone shape from the camera position of the wearable camera HC toward the user's palm side.

[0230] When shooting within the R1 range using the wearable camera HC, the following results are obtained: Figure 24 Image G1 is shown at the center. For example, image G1 includes the DIP joint and the fingertip of the finger near the user's fingertip. The sensor information processing device 30 extracts the positions of the finger joints and fingertips in the image based on the image information acquired from the wearable camera HC as feature points of the finger.

[0231] In addition, the wearable camera HC uses a normal camera or depth sensor to capture images of the user's palm side. An infrared light source can be attached around the camera of the wearable camera HC. The camera can be replaced by a time-of-flight (TOF) sensor. Furthermore, the posture of the wearable camera HC itself is estimated using sensing data from an IMU sensor attached to the same location as the camera.

[0232] As mentioned above, the wearable camera HC can supplement information about fingers that cannot be captured by cameras attached to the environment by photographing the palm side. Furthermore, by photographing the palm side with the wearable camera HC, fingertips can be tracked without being hidden by other fingers.

[0233] [3.6. Structure of Wearable Cameras]

[0234] Next, we will refer to Figure 25 The structure of a wearable camera according to a third embodiment of the present disclosure is described. Figure 25 This is a diagram illustrating the structure of a wearable camera according to a third embodiment of this disclosure.

[0235] like Figure 25 As shown, the wearable camera HC includes a camera C4 that functions as either a normal camera or a depth sensor. It should be noted that because the wearable camera HC is attached to the wrist and photographs the palm, the camera C4 needs to protrude from the watch strap.

[0236] The wearable camera HC includes an IMU sensor (IMU4). The IMU sensor (IMU4) is attached inside the main body of the wearable camera HC.

[0237] In addition, the wearable camera HC includes a strap B1 for securing it to the wrist.

[0238] In addition, the wearable camera HC may include a marker MR1 for tracking from an external sensor surrounding the strap.

[0239] [3.7. Variation]

[0240] Next, refer to Figure 26 This describes the operation of an information processing system modified according to a third embodiment of the present disclosure. Figure 26 This is a diagram illustrating an operational example of an information processing system modified according to a third embodiment of this disclosure. Figure 26 The following describes an implementation of the information processing system 3 that estimates time-series information about finger postures based on image information from a wearable camera and image information from a high-speed camera installed in the environment, without using sensing data from the IMU sensor of the wearable camera.

[0241] exist Figure 26 In the example shown, similar to Figure 23 The information processing device 100B acquires sensor images 1, 2, 3, ... captured by multiple high-speed cameras installed in the environment. Subsequently, the information processing device 100B inputs the acquired sensor images 1, 2, 3, ... into a machine learning model M1. The information processing device 100B estimates the two-dimensional positions of feature points of finger joints, palms, backs of hands, and wrists included in each of the sensor images 1, 2, 3, ... as the output information of the machine learning model M1. Furthermore, the information processing device 100B acquires the camera parameters of each of the multiple high-speed cameras.

[0242] Furthermore, the information processing device 100B estimates the pose of the wearable camera based on the acquired sensor images 1, 2, 3, ... . Subsequently, the information processing device 100B estimates the camera parameters of the wearable camera based on the estimated pose of the wearable camera.

[0243] Subsequently, the information processing device 100B estimates the three-dimensional positions of feature points of finger joints, palm, back of hand, and wrist based on information about camera parameters of the wearable camera, information about the two-dimensional positions of feature points of fingers included in the images captured by the wearable camera, and estimation information about the two-dimensional positions of feature points of finger joints, with the palm, back of hand, and wrist included in the moving images of each camera.

[0244] [4. Fourth Implementation Method]

[0245] In the information processing system 4 according to the fourth embodiment, a contact sensor for detecting contact with an object is installed inside the object. Then, the information processing device 100C of the information processing system 4 according to the fourth embodiment estimates time-series information of the finger's posture in contact with the object based on sensing data about the contact between the finger and the object.

[0246] [4.1. Configuration Example of an Information Processing System]

[0247] Next, refer to Figure 27 The configuration of the information processing system according to the fourth embodiment of this disclosure is described. Figure 27 This is a schematic diagram illustrating a configuration example of an information processing system according to a fourth embodiment of the present disclosure. Figure 27 As shown, the information processing system 4 according to the fourth embodiment differs from the information processing system 1 according to the first embodiment in that it includes a sensor information processing device 40. Furthermore, the information processing system 4 according to the fourth embodiment differs in that it includes an information processing device 100C instead of the information processing device 100 of the information processing system 1 according to the first embodiment. Therefore, in the following description, the sensor information processing device 40 will be described primarily, and detailed descriptions of other configurations included in the information processing system 4 according to the fourth embodiment will be omitted.

[0248] The sensor information processing device 40 acquires sensing data about the contact between a finger and the object from a contact sensor installed within the object. When acquiring sensing data about the contact between a finger and the object, the sensor information processing device 40 sends the sensing data to the information processing device 100C.

[0249] The information processing device 100C acquires sensing data about the contact between a finger and an object from the sensor information processing device 40. Based on the sensing data, the information processing device 100C estimates time-series information about the posture of the finger in contact with the object. It should be noted that the sensor information processing device 40 and the information processing device 100C can be integrated. In this case, the information processing device 100C acquires sensing data about the contact between a finger and the object from a contact sensor installed inside the object.

[0250] [4.2. Operational Example of an Information Processing System]

[0251] Next, refer to Figure 28 The operation of the information processing system according to the fourth embodiment of this disclosure is described. Figure 28 This is a diagram illustrating an operational example of an information processing system according to a fourth embodiment of the present disclosure. Figure 28 In the example shown, similar to the information processing apparatus according to the first to third embodiments, the information processing apparatus 100C estimates the three-dimensional position of the feature points of the finger joints, palm, back of hand and wrist included in the moving image of each camera based on information about the two-dimensional position of the feature points of the finger joints, palm, back of hand and wrist included in the moving image of each camera.

[0252] Furthermore, the information processing device 100C acquires contact information of the fingers with respect to the object from the sensor information processing device 40. Subsequently, the information processing device 100C estimates the fingers that have made contact with the object based on the three-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist, as well as the contact information between the fingers and the object. Additionally, the information processing device 100C acquires a model of the finger used to specify the finger making contact with the object. Subsequently, the information processing device 100C estimates the posture of the finger making contact with the object based on the estimated finger making contact with the object and the acquired finger model.

[0253] [4.3. Example of Information Processing Device Configuration]

[0254] Next, we will refer to Figure 29 The configuration of the information processing apparatus according to the fourth embodiment of this disclosure is described. Figure 29 A diagram illustrating an example configuration of an information processing apparatus according to a fourth embodiment of the present disclosure. (See diagram for reference.) Figure 29 As shown, the information processing apparatus 100C according to the fourth embodiment differs from the information processing apparatus 100 according to the first embodiment in that it provides an estimation unit 132C and a sensor database 121C instead of an estimation unit 132 and a sensor database 121. Therefore, in the following description, the estimation unit 132C and the sensor database 121C will be described primarily, and detailed descriptions of other configurations included in the information processing apparatus 100C according to the fourth embodiment will be omitted.

[0255] (Sensor Database 121C)

[0256] The sensor database 121C differs from the sensor database 121 of the information processing apparatus 100 according to the first embodiment in that it stores sensing data about the contact between a finger and an object, acquired from the sensor information processing apparatus 40. The sensor database 121C stores sensing data about the contact between a finger and an object, acquired by the acquisition unit 131.

[0257] (Estimation Unit 132C)

[0258] The estimation unit 132C estimates time-series information about the posture of the finger in contact with the object based on sensing data detected by a contact sensor that detects the contact operation of the finger relative to the object. Specifically, the estimation unit 132C acquires the contact information of the finger with respect to the object from the sensor information processing device 40. Subsequently, the estimation unit 132C estimates the finger that has made contact with the object based on the three-dimensional positions of feature points of the finger joints, palm, back of hand, and wrist, as well as the contact information of the finger relative to the object. In addition, the estimation unit 132C acquires a model of the finger that is in contact with the object. Subsequently, the estimation unit 132C estimates information about the posture of the finger in contact with the object based on the estimated finger in contact with the object and the acquired finger model. For example, the estimation unit 132C estimates the joint angle of the finger in contact with the object as information about the posture of the finger in contact with the object. Note that, as will be discussed later... Figure 31 The estimation process of the finger joint angle performed by the estimation unit 132C is described in detail.

[0259] [4.4. Finger contact operation relative to an object]

[0260] Next, we will refer to Figure 30 This describes a finger contact operation relative to an object according to a fourth embodiment of the present disclosure. Figure 30 This is a diagram illustrating a finger's contact operation relative to an object according to a fourth embodiment of this disclosure. Figure 30 In the example shown, object O2 is, for example, the keyboard of a piano. A contact sensor FS, which detects contact with the object, is installed inside object O2. Figure 30 In the process, when the index finger of the performer's hand H1 contacts the object O2 at point P1 on the upper surface of the object O2, the contact sensor FS detects the contact between the index finger and the object O2. When the contact between the index finger and the object O2 is detected, the contact sensor FS transmits the contact information between the object O2 and the index finger to the sensor information processing device 40.

[0261] [4.5. Processing for estimating finger joint angles]

[0262] Next, we will refer to Figure 31 The method for estimating the joint angle of a finger according to the fourth embodiment of this disclosure is described. Figure 31 This is a diagram used to describe the estimation process of the joint angle of the finger according to the fourth embodiment of this disclosure. Figure 31 The example shown illustrates a user's finger pressing point P1 on the upper surface of object O3. For instance, when the user's finger presses point P1 at one end of the keyboard, the end of the keyboard closer to the pressing position P1 lowers, and the end of the keyboard further away from the pressing position P1 rises, and thus, the position of object O3, which is the keyboard, changes. Figure 31 In the diagram, the position of object O3 before the finger's contact operation with object O3 is represented by a dashed line. Conversely, the position of object O3 during the finger's contact operation with object O3 is represented by a solid line.

[0263] The estimation unit 132 estimates time-series information about the posture of the finger contacting the object based on the object's position information before the finger-to-object contact operation is performed, the change in the object's position before and after the finger-to-object contact operation, and the finger's contact position information relative to the object. Figure 31 In the process, the estimation unit 132 estimates the time series information about the posture of the finger in contact with the object O3 based on the position information of the object before the finger contact operation with the object O3 (position information of the dashed line), the change in the position of the object before and after the finger contact operation with the object O3 (the change in position between the dashed line and the solid line), and the information of the contact position P1 of the finger with the object O3.

[0264] More specifically, estimation unit 132 estimates the angle of the finger's PIP joint based on the distance between the MP and PIP joints, the distance between the PIP joint and the fingertip, the position of the MP joint, and the position of the fingertip, as time-series information about the finger's posture in contact with the object. Figure 31 In this calculation, estimation unit 132 estimates the angle of the finger's PIP joint based on the distance L1 between the position P3 of the finger's MP joint and the position P2 of the PIP joint, the distance L2 between the position P2 of the finger's PIP joint and the position P1 of the fingertip, and the position P3 of the finger's MP joint and the position P1 of the fingertip. For example, estimation unit 132 estimates the positions P3 of the finger's MP joint, P2 of the PIP joint, and the position P1 of the fingertip, as included in the image information from a high-speed camera mounted in the environment. Then, estimation unit 132 calculates the distance L1 between the position P3 of the finger's MP joint and the position P2 of the PIP joint, and the distance L2 between the position P2 of the finger's PIP joint and the position P1 of the fingertip. Subsequently, estimation unit 132 uses the law of cosines to estimate the angle of the finger's PIP joint based on the calculated distances L1 and L2, the estimated position P3 of the MP joint, and the estimated position P1 of the fingertip. Note that the finger's DIP joint moves synchronously with the finger's PIP joint and is therefore omitted in the calculation.

[0265] [5. Effects]

[0266] As described above, the information processing apparatus 100 according to embodiments of the present disclosure or variations thereof includes an estimation unit 132. The estimation unit 132 estimates time-series information about finger posture based on image information including an object and finger operations relative to the object (including finger contact operations relative to the object). Furthermore, the estimation unit 132 estimates the time-series information about finger posture using a first machine learning model, wherein the first machine learning model is trained to estimate the time-series information about finger posture based on image information including object and finger operations.

[0267] As a result, the information processing device 100 can estimate finger posture without installing sensors or markers on finger joints or the like. That is, the information processing device 100 can estimate finger posture by installing sensors, markers, etc., without hindering finger operation. Therefore, the information processing device 100 can appropriately estimate finger posture during finger operation relative to an object (including finger contact operation relative to an object, such as fingers during piano playing).

[0268] In addition, the estimation unit 132 estimates the time series information of the position, velocity, acceleration or trajectory of each joint of the finger or each feature point of the fingertip, palm, back of the hand or wrist as time series information about the finger posture, or the angle, angular velocity or angular acceleration of each joint of the finger.

[0269] Therefore, the information processing device 100 can not only appropriately estimate the three-dimensional position of the finger, but also appropriately estimate the angle of the finger joint, so that the finger posture can be estimated more appropriately.

[0270] Image information is image information captured by a high-speed monochrome camera or a high-speed infrared camera.

[0271] Therefore, even when the shutter speed is increased to capture high-speed finger movements, the information processing device 100 can ensure sufficient light without causing glare to the user performing the finger movements, and thus, the finger posture can be properly estimated.

[0272] Furthermore, the image information is acquired by each of multiple cameras mounted to capture images of the object from multiple different directions.

[0273] Therefore, when shooting from one direction, the information processing device 100 can cover a finger hidden by another finger or the like by shooting from another direction, and thus, the finger posture can be estimated more appropriately.

[0274] In addition, multiple cameras are attached to a gate-like structure surrounding the object, and each of the multiple image information is multiple image information captured by the finger while it is illuminated by a light source installed near each camera.

[0275] Therefore, even when capturing high-speed finger movements, the information processing device 100 is able to capture images with sufficient light, and thus can more accurately estimate the finger posture.

[0276] Image information consists of multiple images captured by three or more cameras mounted on either side of and above the object.

[0277] Therefore, when shooting from one direction, the information processing device 100 can cover a finger hidden by another finger or the like by shooting from another direction, and thus, the finger posture can be estimated more appropriately.

[0278] In addition, the image information is captured using the area from the fingertip to the wrist as the shooting range.

[0279] Therefore, the information processing device 100 can improve the resolution and accuracy of finger posture estimation by narrowing the shooting range, so that the finger posture can be estimated more appropriately.

[0280] Furthermore, estimation unit 132 estimates time-series information about finger posture based on image information of the back of the hand performing the finger operation. Additionally, estimation unit 132 estimates time-series information about finger posture using a second machine learning model, wherein the second machine learning model is trained to estimate time-series information about finger posture based on image information of the back of the hand performing the finger operation.

[0281] Therefore, the information processing device 100 can more appropriately estimate the finger posture based on an image of the back of the hand that is easier to capture than the fingers during high-speed operation.

[0282] In addition, the estimation unit 132 estimates time-series information about the user's finger posture based on sensing data detected by multiple IMU sensors installed on the thumb and back of the user's hand.

[0283] Therefore, the information processing device 100 can supplement the posture estimation of a finger that is hidden by another finger or the like.

[0284] In addition, the estimation unit 132 estimates time-series information about the user's finger posture based on image information captured by a wearable camera attached to the user's wrist.

[0285] Therefore, the information processing device 100 can supplement the posture estimation of hidden fingers such as other fingers.

[0286] In addition, the wearable camera also includes an IMU sensor, and the estimation unit 132 estimates time-series information about finger posture based on sensing data detected by the IMU sensor.

[0287] Therefore, the information processing device 100 can more accurately supplement the posture estimation of hidden fingers such as other fingers.

[0288] Furthermore, the estimation unit 132 estimates time-series information about the posture of the finger in contact with the object based on sensing data detected by a contact sensor that detects the contact operation of the finger relative to the object. Additionally, the estimation unit 132 estimates time-series information about the posture of the finger in contact with the object based on the object's position information before the finger-to-object contact operation is performed, the change in the object's position before and after the finger-to-object contact operation, and the finger's contact position information relative to the object. Furthermore, the estimation unit 132 estimates the angle of the finger's PIP joint as time-series information about the posture of the finger in contact with the object based on the distance between the MP joint and PIP joint of the finger, the distance between the PIP joint and the fingertip, the position of the MP joint of the finger, and the position of the fingertip.

[0289] Therefore, the information processing device 100 can supplement the posture estimation of a finger that is hidden by another finger or the like.

[0290] Furthermore, the object is the keyboard, and the operation of the finger relative to the object is either a keystroke operation of the finger relative to the keyboard or a movement operation of the finger relative to the position of the keyboard.

[0291] As a result, the information processing device 100 can appropriately estimate the finger posture during piano playing.

[0292] In addition, the information processing device 100 also includes a providing unit 133. The providing unit 133 provides the user with time-series information about finger posture estimated by the estimation unit 132.

[0293] Therefore, the information processing device 100 can transmit fine finger movements to another person (e.g., a student) and support that person's skill level.

[0294] [6. Hardware Configuration]

[0295] For example, by having such Figure 32 The computer 1000 configured as shown implements, for example, the information processing apparatus 100 according to the above embodiments and variations. Figure 32This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of an information processing device, such as information processing apparatus 100. Hereinafter, the information processing apparatus 100 according to the above-described embodiment or its variations will be described as an example. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, hard disk drive (HDD) 1400, communication interface 1500, and input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0296] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 develops programs stored in ROM 1300 or HDD 1400 in RAM 1200 and executes processing corresponding to various programs.

[0297] ROM 1300 stores boot programs, such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 is activated, and programs that depend on the hardware of computer 1000.

[0298] HDD 1400 is a computer-readable recording medium that non-transitoryly records a program executed by CPU 1100, data used by the program, etc. Specifically, HDD 1400 is a recording medium that records an information processing program or a variation thereof according to an embodiment of the present disclosure, which is an example of program data 1350.

[0299] Communication interface 1500 is an interface for computer 1000 to connect to external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device or sends data generated by CPU 1100 to another device via communication interface 1500.

[0300] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as keyboard and mouse via input / output interface 1600. Furthermore, CPU 1100 transmits data to output devices such as monitor, speaker, or printer via input / output interface 1600. Additionally, input / output interface 1600 can be used as a media interface for reading programs, etc., recorded on a predetermined recording medium (medium). For example, the medium is an optical recording medium such as a Digital Universal Disc (DVD) or Phase Change Rewritable Disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), magnetic tape, magnetic recording media, semiconductor memory, etc.

[0301] For example, when the computer 1000 is used as an information processing device 100 according to the above-described embodiments or variations thereof, the CPU 1100 of the computer 1000 implements the functions of the control unit 130, etc., by executing an information processing program loaded onto the RAM 1200. Furthermore, the HDD 1400 stores the information processing program and data according to the embodiments of this disclosure or variations thereof in the storage unit 120. Note that the CPU 1100 reads program data 1350 from the HDD 1400 and executes the program data 1350; however, as another example, these programs can be obtained from another device via an external network 1550.

[0302] It should be noted that this technology may also have the following configurations. (1)

[0304] An information processing device, comprising:

[0305] The estimation unit estimates time-series information about finger pose based on image information, which includes: the object and the finger's operation relative to the object, including the finger's contact operation relative to the object. (2)

[0307] According to the information processing device in (1),

[0308] The estimation unit estimates time-series information about finger postures by using a first machine learning model, which learns to estimate time-series information about finger postures based on image information including finger manipulation and objects. (3)

[0310] According to the information processing device of (1) or (2),

[0311] The estimation unit estimates the position, velocity, acceleration, or trajectory of each joint of the finger or feature point of each fingertip, palm, back of hand, or wrist, or the angle, angular velocity, or angular acceleration of each joint of the finger as time-series information about the finger's posture. (4)

[0313] The information processing device according to any one of (1) to (3),

[0314] The image information is captured by a high-speed monochrome camera or a high-speed infrared camera. (5)

[0316] Information processing apparatus according to any one of (1) to (4),

[0317] The image information is obtained from multiple images captured by multiple cameras installed to photograph the object from multiple different directions. (6)

[0319] According to the information processing device in (5),

[0320] Multiple cameras are attached to a gate-like structure surrounding the object; and

[0321] Each of the multiple image messages is a series of images captured while the finger is illuminated by a light source installed near each camera. (7)

[0323] The information processing device according to any one of (1) to (6),

[0324] The image information consists of multiple images captured by three or more cameras mounted on either side of the object and above the object. (8)

[0326] Information processing apparatus according to any one of (1) to (7),

[0327] The image information refers to images captured within a range from the fingertip to the wrist. (9)

[0329] The information processing device according to any one of (1) to (8),

[0330] The estimation unit estimates time-series information about finger posture based on image information of the back of the hand performing the finger operation. (10)

[0332] According to the information processing device of (9),

[0333] The estimation unit estimates time-series information about finger posture by using a second machine learning model. The second machine learning model learns to estimate time-series information about finger posture based on image information of the back of the hand performing the finger operation. (11)

[0335] The information processing apparatus according to any one of (1) to (10),

[0336] The estimation unit estimates time-series information about the user's finger posture based on sensing data detected by multiple IMU sensors installed on the thumb and back of the user's hand. (12)

[0338] Information processing apparatus according to any one of (1) to (11),

[0339] The estimation unit estimates time-series information about the user's finger posture based on image information captured by a wearable camera attached to the user's wrist. (13)

[0341] According to the information processing device in (12),

[0342] Wearable cameras also include IMU sensors; and

[0343] The estimation unit estimates time-series information about finger posture based on sensing data detected by the IMU sensor. (14)

[0345] The information processing apparatus according to any one of (1) to (13),

[0346] The estimation unit estimates time-series information about the posture of the finger in contact with the object based on sensing data detected by a contact sensor that detects the finger's contact operation with the object. (15)

[0348] According to the information processing device of (14),

[0349] The estimation unit estimates time-series information about the posture of the finger that is in contact with the object based on the object's position information before the finger-to-object contact operation is performed, the change in the object's position before and after the finger-to-object contact operation is performed, and the finger's contact position information relative to the object. (16)

[0351] According to the information processing device of (14) or (15),

[0352] The estimation unit estimates the angle of the PIP joint of the finger based on the distance between the MP joint and PIP joint of the finger, the distance between the PIP joint and the fingertip, the position of the MP joint of the finger, and the position of the fingertip, as time-series information about the posture of the finger in contact with the object. (17)

[0354] The information processing apparatus according to any one of (1) to (16),

[0355] The object is the keyboard; and

[0356] Finger operations relative to an object are either keystrokes or movement operations involving the finger relative to the keyboard. (18)

[0358] The information processing apparatus according to any one of (1) to (17) further includes:

[0359] The providing unit is configured to provide the user with time-series information about finger posture estimated by the estimation unit. (19)

[0361] An information processing method, comprising:

[0362] This allows a computer to estimate time-series information about finger poses based on image information, including: an object and finger operations relative to the object, such as finger-to-object contact operations. (20)

[0364] A program that enables a computer to be used as an estimation unit to estimate time-series information about finger poses based on image information, the image information including: an object and an operation of the finger relative to the object, the operation of the finger relative to the object including a contact operation of the finger relative to the object.

[0365] Reference number list

[0366] 1. Information Processing System

[0367] 10 Sensor Information Processing Device

[0368] 100 Information Processing Device

[0369] 110 Communication Unit

[0370] 120 storage units

[0371] 121 Sensor Database

[0372] 122 Model Database

[0373] 123 Three-Dimensional Feature Database

[0374] 130 Control Unit

[0375] 131 Acquisition Unit

[0376] 132 Estimation Units

[0377] 133 Providing Unit

[0378] 200 Application Server

[0379] 300 terminal devices.

Claims

1. An information processing apparatus, comprising: The estimation unit estimates time-series information about finger posture based on image information, which includes: an object and the finger's operation relative to the object, the finger's operation relative to the object including contact operations between the finger and the object, and the image information is acquired by multiple cameras mounted to capture images of the object from multiple different directions. The estimation unit is further configured to estimate time-series information about the posture of the user's fingers based on sensing data detected by multiple IMU sensors mounted on the thumb and back of the user's hand. In cases where it is determined that the feature points of the finger are not included in the motion images of each of the plurality of cameras, the estimation unit estimates the position of the feature points of the finger based on information about the relative pose between the plurality of IMU sensors and information about the model of the finger, wherein the relative pose is determined based on the sensing data acquired from each of the plurality of IMU sensors.

2. The information processing device according to claim 1, in, The estimation unit estimates the time-series information about the posture of the finger by using a first machine learning model, the first machine learning model being trained to estimate the time-series information about the posture of the finger based on image information including the operation of the finger and the object.

3. The information processing device according to claim 1, in, The estimation unit estimates the position, velocity, acceleration, or trajectory of each joint of the finger or each fingertip, palm, back of hand, or wrist feature point, or the time-series information of the angle, angular velocity, or angular acceleration of each joint of the finger, as the time-series information about the posture of the finger.

4. The information processing device according to claim 1, in, The image information is image information captured by a high-speed monochrome camera or a high-speed infrared camera.

5. The information processing device according to claim 1, in, Multiple cameras are attached to a gate-like structure surrounding the object; and The image information is the image information captured while the finger is illuminated by a light source installed near each of the cameras.

6. The information processing apparatus according to claim 1, in, The image information is image information captured by three or more cameras mounted on both sides and above the object.

7. The information processing apparatus according to claim 1, in, The image information is image information captured from the fingertip to the wrist.

8. The information processing apparatus according to claim 1, in, The estimation unit estimates the time-series information about the posture of the finger based on image information of the back of the hand performing the operation of the finger.

9. The information processing apparatus according to claim 8, in, The estimation unit estimates the time-series information about the posture of the finger by using a second machine learning model, the second machine learning model being trained to estimate the time-series information about the posture of the finger based on the image information of the back of the hand performing the operation of the finger.

10. The information processing apparatus according to claim 1, in, The estimation unit estimates the time-series information about the posture of the user's fingers based on the image information captured by a wearable camera attached to the user's wrist.

11. The information processing apparatus according to claim 10, in, The wearable camera also includes an IMU sensor; and The estimation unit estimates the time-series information about the finger's posture based on sensing data detected by the IMU sensor.

12. The information processing apparatus according to claim 1, in, The estimation unit estimates the time-series information about the posture of the finger in contact with the object based on sensing data detected by a contact sensor that detects the contact operation of the finger relative to the object.

13. The information processing apparatus according to claim 12, in, The estimation unit estimates the time-series information about the posture of the finger in contact with the object based on the object's position information before the contact operation between the finger and the object is performed, the change in the object's position before and after the contact operation between the finger and the object, and the contact position information between the finger and the object.

14. The information processing apparatus according to claim 12, in, The estimation unit estimates the angle of the PIP joint of the finger based on the distance between the MP joint and PIP joint of the finger, the distance between the PIP joint of the finger and the fingertip, the position of the MP joint of the finger, and the position of the fingertip of the finger, as the time-series information about the posture of the finger in contact with the object.

15. The information processing apparatus according to claim 1, in, The object is a keyboard; and The operation of the finger relative to the object is either a keystroke operation of the finger relative to the keyboard or a movement operation of the finger relative to the position of the keyboard.

16. The information processing apparatus according to claim 1, further comprising: A providing unit is configured to provide the user with the time-series information about the posture of the finger estimated by the estimation unit.

17. An information processing method, comprising: This allows a computer to estimate time-series information about finger poses based on image information, including: an object and the finger's actions relative to the object, the finger's actions relative to the object including contact actions relative to the object, and the image information being acquired by multiple cameras mounted to capture images of the object from multiple different directions. Specifically, the time-series information about the posture of the user's fingers is estimated based on sensing data detected by multiple IMU sensors mounted on the thumb and back of the user's hand. Where it is determined that the feature points of the finger are not included in the motion images of each of the plurality of cameras, the position of the feature points of the finger is estimated based on information about the relative pose between the plurality of IMU sensors and information about the model of the finger, wherein the relative pose is determined based on the sensing data acquired from each of the plurality of IMU sensors.

18. A computer-readable storage medium having a program stored thereon, the program, when executed by a computer, causing the computer to function as an estimation unit, the estimation unit estimating time-series information about finger posture based on image information, the image information including: The operation of the object and the finger relative to the object, the operation of the finger relative to the object including contact operation of the finger relative to the object, and the image information being image information acquired by multiple cameras mounted to capture images of the object from multiple different directions. The estimation unit is further configured to estimate time-series information about the posture of the user's fingers based on sensing data detected by multiple IMU sensors mounted on the thumb and back of the user's hand. In cases where it is determined that the feature points of the finger are not included in the motion images of each of the plurality of cameras, the estimation unit estimates the position of the feature points of the finger based on information about the relative pose between the plurality of IMU sensors and information about the model of the finger, wherein the relative pose is determined based on the sensing data acquired from each of the plurality of IMU sensors.

Images