Finger shape estimation device, finger shape estimation method, and program

Abstract

To provide a finger shape estimation device, a finger shape estimation method, and a program capable of improving mobility and accurately estimating how a person touches an object.SOLUTION: A finger shape estimation device is a device that estimates a shape of a finger of an operator by using a captured image of the finger of the operator, and includes a finger imaging unit that acquires an image of the shape of the finger of the operator from a ventral side of the finger, and a hand posture estimation unit that analyzes the acquired image and estimates a posture of a hand of the operator.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a human hand posture estimating device, a human hand posture estimating method, and a program. [Background technology]

[0002] In recent years, expectations for remote operation of robots and the like have been rising. When remotely operating a robot, for example, a camera is mounted on the head of the robot, and an operator remotely controls the robot by giving operation instructions using a data glove or the like based on the image captured by the camera. In such remote operation, the shape and position of the operator's fingertips and the position of the robot's hand are important. For this reason, development is being conducted on estimating the shape of the fingers and the contact position of the fingertips with the object to be operated.

[0003] For example, in the technology described in Patent Document 1, a camera is attached to the back of the hand and an image of the back of the hand is taken from an upper, forward angle between the thumb and index finger, and the flexion and extension of the thumb, index finger, and middle finger is estimated. Also, for example, a technology has been proposed that compiles a database of places where people touch objects via the Internet and estimates whether or not people have touched the object (see, for example, Non-Patent Document 1). In addition, a technology has been proposed that involves setting up a laboratory with adjusted lighting and installing eight cameras around the worker's hands to estimate the position of the worker's hands. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-100697 A [Non-patent literature]

[0005] [Non-Patent Document 1] Shikhar Bahl, Russell Mendonca, et al., “Affordances from Human Videos as a Versatile Representation for Robotics”, CVPR 2023, 2023 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Non-Patent Document 1 could estimate whether or not an object was in contact with a finger, but could not determine the contact position of each finger with the target object. In addition, with the technology described in Patent Document 1, there were cases where the fingertip was hidden and could not be photographed, such as when the finger was bent. Furthermore, with the conventional technology that uses multiple cameras in a laboratory to photograph, although it was possible to reduce the obstruction of the hand or object, the measurement environment was large and therefore could not be carried around.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a hand posture estimating device, a hand posture estimating method, and a program that can improve mobility and accurately estimate how a person touches an object. [Means for solving the problem]

[0008] (1) In order to achieve the above object, a hand posture estimation device according to one embodiment of the present invention is a device that estimates the posture of an operator's hand by using captured images of the operator's fingers, and is equipped with a hand photographing unit that obtains images of the shape of the operator's fingers from the pad side of the fingers, and a hand posture estimation unit that analyzes the obtained images to estimate the posture of the operator's hand.

[0009] (2) The human hand posture estimating device according to the aspect of the present invention described in (1) may further include a control command generating unit that generates an operation command value for a robot from the estimated hand posture of the operator.

[0010] (3) In the hand posture estimating device according to one aspect of the present invention described in (1), the device may further include a manipulation object capturing unit that captures an image of a manipulation object manipulated by the operator's hand, a manipulation object posture estimation unit that estimates a posture of the manipulation object from the image of the manipulation object, and a contact estimation unit that estimates a contact point and a normal between the manipulation object and the hand from the estimated manipulation object posture and posture of the hand.

[0011] (4) In the hand posture estimating device according to any one of (1) to (3) of the present invention, the hand photographing unit may be installed in a direction to photograph the palm side of the fingers from the palm surface of the operator.

[0012] (5) In a hand posture estimating device according to any one of (1) to (4) of the present invention, the hand photographing unit may include a plurality of photographing units, and a first hand photographing unit may be installed in a direction to photograph the inner side of the fingers from the palm side of the operator's index finger, and the second hand photographing unit may be installed in a direction to photograph the inner side of the fingers from the palm side of the operator's little finger.

[0013] (6) In the hand posture estimating device according to any one of (1) to (3) of the aspects of the present invention, the hand photographing unit may be installed facing in a direction to photograph the thumb from near the ball of the operator's little finger.

[0014] (7) In the hand posture estimating device according to one aspect of the present invention described in (3), the manipulation object capturing unit may be a parallax image capturing camera that generates a parallax image, and the hand posture estimating unit may capture an image from the palm toward a direction other than the back of the hand, and estimate the posture of the hand from the image captured by the manipulation object capturing unit and an image acquired by the hand capturing unit.

[0015] (8) In a hand posture estimation device according to any one of (1) to (7) of the present invention, the device may further include an imaging unit that acquires an image of the shape of the operator's hand from the upper part of the operator's palm, and the hand posture estimation unit may analyze the images acquired by the imaging unit and the hand posture estimation unit to estimate the posture of the operator's hand.

[0016] (9) In order to achieve the above-mentioned object, the present invention provides a method for estimating a hand posture of an operator using a captured image of the operator's hand, the method including: a hand photographing unit acquiring an image of the shape of the operator's fingers from the pad side of the fingers; and a hand posture estimation unit analyzing the acquired image to estimate the posture of the operator's hand.

[0017] (10) In order to achieve the above object, a program is provided that uses a captured image of an operator's hand and fingers in one embodiment of the present invention to cause a computer of a device that estimates the shape of the operator's fingers to obtain an image of the shape of the operator's fingers from the pad side of the fingers, and causes a hand posture estimation unit to analyze the obtained image and estimate the posture of the operator's hand. Effect of the Invention

[0018] According to the above (1) to (10), it is possible to improve mobility and accurately estimate how a person touches an object. [Brief description of the drawings]

[0019] [Figure 1] 1 is an example of a palm camera according to an embodiment attached to a palm. [Diagram 2] 1 is a diagram illustrating an example of the configuration of a human hand posture estimating system according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing an overview of a remote operation space and a robot working space according to the first embodiment. [Figure 4] 1A to 1C are conceptual diagrams showing example images captured by each camera. [Diagram 5] 3A to 3C are diagrams illustrating an example of the shooting range of each camera according to the first embodiment. [Figure 6] 3A and 3B are diagrams showing an example of a mount according to the first embodiment as viewed from above and from the side. [Figure 7] FIG. 13 is a diagram showing an example of the position and shooting range of each palm camera when the palm cameras are attached to the prototype mount. [Figure 8] 1A to 1C are diagrams showing examples of hand positions, fingertip positions, and finger postures. [Figure 9] 4A to 4C are diagrams illustrating an example of the configuration and processing of a functional section that processes information from a head camera according to an embodiment. [Figure 10] 5A to 5C are diagrams illustrating an example of processing contents and a processing procedure performed by the control device according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing a schematic configuration of the process shown in FIG. 9 using a network. [Figure 12] 1A and 1B are diagrams showing an example of a prototype mount and an example of the mount being worn on a hand. [Figure 13] 13A and 13B are diagrams illustrating other examples of camera positions in the palm camera that place emphasis on fingertip contour matching processing. [Figure 14] 13A and 13B are diagrams illustrating other examples of camera positions in a palm camera that prioritizes distance measurement. [Figure 15] FIG. 11 is a diagram illustrating an example of the configuration of a human hand posture estimating system according to a second embodiment. [Figure 16] FIG. 13 is a diagram illustrating an example of input and output to a model during learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size. In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted. In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0021] (Palm Camera) First, an example of the palm camera 2 of the embodiment will be described. FIG. 1 shows an example of a palm camera according to this embodiment attached to a palm. In the example of FIG. 1, palm camera 2 is attached to the inside of the palm. Palm first camera 21 is placed, for example, near the proximal phalanx of the index finger (including the thenar eminence) and between the index finger and the thumb. Palm second camera 22 is placed, for example, near the proximal phalanx of the little finger (including the thenar eminence) and on the outside, not on the ring finger side. Palm third camera 23 is installed, for example, near the thenar eminence of the palm. Palm fourth camera 24 is placed, for example, near the proximal phalanx of the ring finger (including the thenar eminence). Palm fifth camera 25 is placed, for example, near the proximal phalanx of the middle finger (including the thenar eminence).

[0022] Palm-first camera 21 to palm-third camera 23 are attached to mount 27 made of, for example, an elastic material. Palm-first camera 21 to palm-third camera 23 capture images of the hand and other parts from a precision grip to a strength grip. Palm fourth camera 24 and palm fifth camera 25 are fixed to mount 28 made of a material that is difficult to deform so that the distance between the cameras does not change. Palm fourth camera 24 and palm fifth camera 25 are RGB (red-green-blue) stereo cameras. In this embodiment, depth information is estimated using images captured by the stereo cameras.

[0023] In the case of the camera configuration and arrangement as shown in Fig. 1, it is preferable that the cameras required for photographing the fingers in palm camera 2 are palm-first camera 21 to palm-third camera 23. The reason is that palm-first camera 21 can photograph the index finger and middle finger, while palm-second camera 22 can photograph the ring finger and little finger. Palm-first camera 21 and palm-second camera 22 can also photograph the thumb depending on its state, but the thumb has a wide range of motion. For this reason, it is preferable to provide palm-third camera 23 in order to photograph the thumb when it is about to grasp or when it is actually grasped. However, depending on the size of the object and the work being done, it may be possible to capture an image of the thumb with palm-first camera 21 and palm-second camera 22. In such cases, palm camera 2 does not need to be equipped with palm-third camera 23. Furthermore, depending on the work, palm camera 2 may be equipped with one of palm-first camera 21 to palm-third camera 23, or one of the three may be used.

[0024] The arrangement of the cameras and the material of the mount 27 described with reference to FIG. 1 are merely examples and are not limited to these.

[0025] As a result of the camera arrangement as shown in FIG. 1, according to this embodiment, palm camera 2 moves together with the palm in response to work instructions, and the fingers do not collide with the camera when the hand assumes various postures. In addition, the shape and position of the fingertips can be detected with high accuracy when the operator grasps the object precisely during operation. In this embodiment, interference with an object can be checked, particularly with a precision grip, without the palm of the hand interfering with imaging.

[0026] If the camera were placed on the pad of the finger, it would be impossible to capture an image of the fingertips when gripping the device, and even if the camera were placed in the center of the palm or on the wrist, it would be impossible to capture an image of the fingertips when, for example, striking a piano key. For this reason, in this embodiment, the camera is placed as shown in Fig. 1 as an example, so that it is possible to properly capture an image of the fingertips when operating the device. Furthermore, the cameras do not necessarily have to be placed on the inside of the hand, but may be on the side or back of the palm. However, even in such a placement, palm camera 2 should be placed so that it can capture the inside of the hand, that is, so that it can capture the shape of the fingers and the position of the fingertips without being hidden.

[0027] First Embodiment In this embodiment, when an operator wears a palm camera 2 and a head camera 3 and remotely controls a robot 5, the shape of the operator's hand and the like are estimated. First, a hand posture estimating system 1 of this embodiment will be described. Fig. 2 is a diagram showing an example of the configuration of a human hand posture estimation system according to this embodiment. As shown in Fig. 2, the human hand posture estimation system 1 includes, for example, a palm camera 2 (a hand image capturing unit, an operation object image capturing unit), a head camera 3 (image capturing unit), a control device 4 (a human hand posture estimation device), a robot 5, an environment sensor 6, and an HMD 7.

[0028] Palm camera 2 includes, for example, palm first camera 21 (finger photographing section), palm second camera 22 (finger photographing section), palm third camera 23 (finger photographing section), palm fourth camera 24 (operating object photographing section), palm fifth camera 25 (operating object photographing section), and communication section 26. The head camera 3 includes, for example, a camera 31 and a communication unit 32 . The control device 4 includes, for example, a first acquisition unit 41, a second acquisition unit 42, a hand posture estimation unit 43, an operating object posture estimation unit 44, a contact estimation unit 45, a control command generation unit 46, an output unit 47, a third acquisition unit 48, and an image generation unit 49. The robot 5 includes, for example, an end effector 51, an arm 52, a driving unit 53, a sensor 54, an acquisition unit 55, and an output unit 56. The environmental sensor 6 includes, for example, a camera 61 and a communication unit 62.

[0029] The palm camera 2 and the control device 4 are connected by a wireless or wired network NW. The head camera 3 and the control device 4 are connected by a wireless or wired network NW. The control device 4 and the robot 5 are connected by a wireless or wired network NW. The control device 4 and the environmental sensor 6 are connected by a wireless or wired network NW. The control device 4 and the HMD 7 are connected by a wireless or wired network NW.

[0030] Palm camera 2 is attached to the palm of an operator's hand, for example, on the inside of the palm. Each of the first palm camera 21 to the fifth palm camera 25 is, for example, an imaging device using a CCD (Charge Coupled Device) imaging element or a CMOS (Complementary Metal Oxide Semiconductor) imaging element. The first palm camera 21 to the third palm camera 23 capture, for example, an image of the shape of the operator's fingers from the pad side of the operator's fingers. The fourth palm camera 24 to the fifth palm camera 25 capture, for example, an image of an object operated by the operator's hand. The communication unit 26 transmits image data captured by each of the palm-first camera 21 to palm-fifth camera 25 to the control device 4.

[0031] The head camera 3 is attached to, for example, the head of the operator. The camera 31 is, for example, a photographing device using a CCD image sensor or a CMOS image sensor. The communication unit 32 transmits image data captured by the camera 31 to the control device 4.

[0032] The HMD 7 is, for example, a head mounted display. The HMD 7 includes, for example, an image display unit and a communication unit. The image display unit displays an image acquired from the control device 4 via the communication unit. The operator performs remote control while viewing the image displayed on the HMD 7. The head camera 3 may be attached to the HMD 7.

[0033] The robot 5 includes at least one arm and an end effector. The robot 5 may be a humanoid robot including a body, a head, and the like, or may be a bipedal robot capable of walking on two legs. The robot 5 also includes a power supply unit (not shown) and the like.

[0034] The end effector 51 is, for example, a gripper or a multi-fingered hand, and may have three or more fingers. Sensors 54 are attached to the joints, finger pads, etc. of the end effector 51. Also, the end effector 51 has an actuator for each joint.

[0035] An end effector 51 is attached to the tip of the arm 52. The opposite side of the arm 52 is attached to, for example, a body. A sensor 54 is attached to the joint of the arm 52. Also, the arm 52 has an actuator for each joint.

[0036] The drive unit 53 drives the arm 52 and the end effector 51 in accordance with the operation command value acquired by the acquisition unit 55 from the control device 4.

[0037] The sensor 54 is a sensor such as an encoder attached to each joint of the end effector 51 and the arm 52. The sensor 54 is a pressure detection sensor attached to the pad of a finger of the end effector 51, for example.

[0038] The acquisition unit 55 acquires the operation command value output by the control device 4.

[0039] The output unit 56 outputs, for example, a detection result detected by the sensor 54 to the control device 4. The output unit 56 outputs an image generated by the image generating unit to the HMD 7.

[0040] The environmental sensor 6 is installed in the working space of the robot 5. There may be two or more environmental sensors 6. Furthermore, the robot 5 may be equipped with the environmental sensor 6. The camera 61 is, for example, an RGBD camera that can also obtain depth information D. The camera 61 may be an RGB camera and a distance sensor, or may be a stereo camera. The communication unit 62 transmits image data captured by the camera 61 to the control device 4.

[0041] The first acquisition unit 41 acquires the image data output by the palm camera 2.

[0042] The second acquisition unit 42 acquires the image data output by the head camera 3 .

[0043] The third acquisition unit 48 acquires image data output by the environmental sensor 6. The third acquisition unit 48 acquires the detection result of the sensor 54 output by the robot 5, and the like.

[0044] The hand posture estimation unit 43 estimates the distance from the hand and the labels of the parts of the hand using the image data acquired by the second acquisition unit 42. Then, the hand posture estimation unit 43 converts the estimation result into the initial shape and posture of the fingers, the labels of each finger, and a point cloud of the fingers. Note that a point cloud is a collection of points in a three-dimensional space, and each point has coordinates. Furthermore, the hand posture estimation unit 43 analyzes the image data from the palm first camera 21 to the palm third camera 23 out of the image data acquired by the first acquisition unit 41, and estimates the posture of the operator's hand.

[0045] The manipulation object posture estimation unit 44 estimates the distance from the manipulation object using the image data acquired by the second acquisition unit 42. Then, the manipulation object posture estimation unit 44 converts the estimation result into the initial position of the manipulation object. Moreover, the manipulation object posture estimation unit 44 estimates the posture of the manipulation object using the image data of the fourth palm camera 24 to the fifth palm camera 25 among the image data acquired by the first acquisition unit 41.

[0046] The contact estimation unit 45 estimates the contact point and the normal of the surface from the position where the surface drawn on the surface of the point cloud intersects, from the point cloud of the operation object and the hand obtained from the posture of the operation object estimated by the operation object posture estimation unit 44 and the posture of the operator's hand estimated by the hand tip posture estimation unit 43. The contact estimation unit 45 may calculate the normal by searching for a representative point with the minimum distance between polygons representing the shapes using a gradient method, based on the results of estimating the hand shape and object shape. As will be described later, the hand posture estimation unit 43, the manipulation object posture estimation unit 44, and the contact estimation unit 45 are a model configured, for example, by a network.

[0047] The control command generator 46 generates an operation command value from the posture of the operator's hand estimated by the hand tip posture estimator 43.

[0048] The output unit 47 outputs the operation command value generated by the control command generation unit 46 to the robot 5.

[0049] (Remote control space and robot workspace) Next, an overview of the remote control space and the robot workspace will be given. FIG. 3 is a diagram showing an overview of the remote operation space and the robot working space according to this embodiment. The image g101 is an example of a remote control space in which an operator remotely controls a robot. In the remote control space, a palm camera 2, a head camera 3, and an HMD 7 are used. Reference symbol g102 denotes a robot workspace in which the robot 5 works. In the robot workspace, the arm 52, the end effector 51, and the environmental sensor 6 of the robot 5 are used.

[0050] When remotely controlling the robot 5, the operator wears the palm camera 2 on his / her palm, and the head camera 3 and HMD 7 on his / her head. The operator remotely controls the robot 5 while looking at the image displayed on the HMD 7. The control device 4 estimates the operator's hand posture, the posture of the object to be operated, and the contact state between the object to be operated (obj) and the hand based on the image data acquired from the palm camera 2 and the head camera 3, and generates an operation command value for the robot 5 based on the estimated result. The robot 5 drives the arm 52 and the end effector 51 based on the operation command to perform the operation. The operation contents include, for example, picking up the object to be operated, grasping the object to be operated, moving the object to be operated, opening the lid of a bottle or the like, tightening a screw, loosening a screw, etc.

[0051] If the mount of the palm camera 2 is made of a contractile material, the control device 4 (e.g., the hand posture estimation unit 43) measures the distance to the object from the image data, and performs SLAM (Simultaneous Localization and Mapping) processing to estimate the self-position and create a map for each camera at a predetermined time. The coordinate system of the camera position and the map may be, for example, the coordinate system of the head camera 3. This makes it possible to grasp the position of each camera of the palm camera 2 even if it shifts during remote control, etc. Furthermore, because of this processing and information, in this embodiment, the image of the head camera 3 can be converted into the coordinates of the wrist.

[0052] Even if the mount of the palm camera 2 is made of a material that does not contract, the control device 4 may perform the above-mentioned processing. This can reduce the influence of the camera position shift when it is attached.

[0053] (Images taken by each camera) FIG. 4 is a conceptual diagram showing example images captured by each camera. Image g201 is an example of an image captured by camera 31 of the head camera 3. Image g211 is an example of an image captured by palm-first camera 21 of palm camera 2. Image g212 is an example of an image captured by second palm camera 22 of palm camera 2. Image g213 is an example of an image captured by palm-third camera 23 of palm camera 2. Image g214 is an example of an image captured by the fourth palm camera 24 of the palm camera 2. Image g215 is an example of an image captured by the fifth palm camera 25 of the palm camera 2.

[0054] Even when the image captured by camera 31 of head camera 3 is an occlusion where an object in the foreground hides an object behind, as in image g201, palm camera 2 can capture the object, as in images g202 to g215. Furthermore, images g212 and g213 taken by palm-fourth camera 24 and palm-fifth camera 25 clearly capture a pair of grasped and assembled objects at the center of the image. Furthermore, in this embodiment, the angles of view of palm-first camera 21, palm-second camera 22, and palm-third camera 23 are adjusted around the fingers (thumb, index finger, and middle finger) that the operator uses most frequently for operations, so that the condition of each finger and the positional relationship between the fingers and the object can be appropriately captured.

[0055] Fig. 5 is a diagram showing an example of the imaging range of each camera according to the embodiment. The example in Fig. 5 is also an example in which the head camera 3 is attached to the HMD 7. Fig. 6 is a diagram showing an example of the mount according to the embodiment as viewed from above and from the side. In Fig. 6, the diagram with reference numeral 350 shows the shape of the mount as viewed from above, the position of each camera, and the imaging range of each camera. The diagram with reference numeral 360 shows the shape of the mount as viewed from the side, and the imaging range of each camera.

[0056] In FIG. 5, a triangular area g301 is an example of the imaging area of ​​the head camera 3. 5 and 6, triangular area g311 is an example of the imaging range of palm-first camera 21 in the vertical direction relative to the palm. Triangular area g312 is an example of the imaging range of palm-second camera 22 in the vertical direction relative to the palm. Triangular area g313 is an example of the imaging range of palm-third camera 23 in the vertical direction relative to the palm. Triangular area g314 is an example of the imaging range of palm-fourth camera 24 in the vertical direction relative to the palm. Triangular area g315 is an example of the imaging range of palm-fifth camera 25 in the vertical direction relative to the palm.

[0057] 5 and 6, the chain-line triangular range g321 is an example of the range captured on the inside of the palm by palm-first camera 21. Palm-first camera 21 captures, for example, the middle finger through index finger. Chain-line triangular range g322 is an example of the range captured on the inside of the palm by palm-second camera 22. Palm-second camera 22 captures, for example, the middle finger through index finger. Chain-line triangular range g323 is an example of the range captured on the inside of the palm by palm-third camera 23. The image captured by palm-third camera 23 is used, for example, to measure the contact state between the thumb and an operation object.

[0058] 5 and 6, the chain-line circle range g331 is an example of the range captured in the inward direction of the palm by palm-fourth camera 24. The chain-line circle range g332 is an example of the range captured in the inward direction of the palm by palm-fifth camera 25. Images captured by palm-fourth camera 24 and palm-fifth camera 25 are used, for example, to detect an object to be operated and to measure the shape of the object to be operated. The optical axes of palm-fourth camera 24 and palm-fifth camera 25 are parallel.

[0059] As shown in the diagram of reference symbol g360 in FIG. 6, the palm camera 2 has a mount (band) 28 made of a contractile material attached to the back of the palm, and the mount 27 is attached to the palm. Note that the range examples shown in FIGS. 5 and 6 are merely examples and are not limiting.

[0060] Figure 7 shows an example of the position and shooting range of each camera when each palm camera is attached to the prototype mount. Image g370 is an image seen from the fingertip. Image g380 is an image seen from the ball of the little finger. In images g370 and g380, reference symbol g371 indicates palm first camera 21, reference symbol g372 indicates palm second camera 22, reference symbol g373 indicates palm third camera 23, reference symbol g314 indicates palm fourth camera 24, reference symbol g315 indicates palm fifth camera 25, and reference symbol g376 indicates mount 27.

[0061] Next, the hand position, fingertip positions, and finger posture in this embodiment will be described. Fig. 8 is a diagram showing an example of the hand position, the fingertip positions, and the finger postures. The example in Fig. 8 shows a state in which an operation object obj is being grasped with the right hand. Point g391 is the position of the hand (position of the wrist). Point g392 is the position of the fingertips (position of the fingers). Line g393 is the posture of the fingers. In this embodiment, such initial positions of the hand and the initial positions of the fingertips are detected using images captured by the head camera 3 and the palm camera 2. Also, in this embodiment, such positions of the hand, the positions of the fingertips, and the postures of the fingers are detected using images captured by the head camera 3 and the palm camera 2.

[0062] In the example shown in FIG. 8, a hand shape is skeletonized by a known method, but the present invention is not limited to this. The control device 4 may generate a mesh around the skeleton based on the skeleton and calibrate the meshes to match them. Alternatively, the control device 4 may directly obtain the initial position of the hand, the initial position of the fingertip, the position of the hand, the position of the fingertip, and the posture of the fingers using images captured by the head camera 3 and the palm camera 2, respectively. For example, the control device 4 may detect the outline of the finger based on the captured image, and reconstruct a photomap from the mesh around the finger based on the detection result.

[0063] In addition, since the image captured by the head camera 3 is projected onto the image captured by the palm camera 2, in this embodiment, the position of the wrist is set as the reference position. In addition, because the head camera 3 is positioned farther away from the palm camera 2, there is a misalignment between the image captured by the palm camera 2 and the image captured by the head camera 3. In this embodiment, this misalignment is corrected as described below.

[0064] The frame (coordinate system) used for calculation is, for example, a frame (coordinate system) in which the initial values ​​of the hand are calculated using the head camera 3. However, the control device 4 needs to process information on both the palm camera 2 and the object to be operated in the same frame. Alternatively, the control device 4 may process the head camera 3, palm camera 2, and object to be operated using coordinates located elsewhere as the origin. That is, in this embodiment, all the cameras are expressed in the same coordinate system.

[0065] (Examples of processing content and processing procedures) Next, an example of the process contents and the process procedure performed by the control device 4 will be described. FIG. 9 is a diagram showing an example of the configuration and processing of a functional section that processes information from the head camera according to this embodiment. The functional unit that processes information from the head camera 3 includes encoders g401 and g403 and a decoder g405, and corresponds to the hand posture estimation unit 43 and the manipulation object posture estimation unit . The first encoder g401 receives as input a point cloud of an object generated based on information output by the head camera 3. The output of the first encoder g401 is feature amounts of the point cloud of the object, and is input to a first hidden layer g407-1 of the decoder g405 and a first input layer g402-1 of the decoder g405. The second encoder g403 receives the hand point cloud and the output of the first encoder g401. The output of the second encoder g403 is a feature obtained by convolving the hand point cloud, and is input to a variational autoencoder (VAE) g404. The variational autoencoder g404 uses a latent vector with a distribution expressed by μ and σ of normal distribution. The variational autoencoder g404 learns μ and σ so that the hand point cloud fits the hand in the real image. The output of the variational autoencoder g404 is input to the second input layer g402-2 of the decoder g405. Three-dimensional (3D) points generated based on information output by the head camera 3 are input to a third input layer g402-3 of the decoder g405.

[0066] The outputs of the first input layer g402-1 to the third input layer g402-3 of the decoder g405 are input to a first convolutional layer g406. The output of the first convolutional layer is input to a second hidden layer g407-2. The outputs of the first hidden layer g407-1 and the second hidden layer g407-2 are input to the second convolutional layer g408. The output of the second convolutional layer g408 is output via the output layer g408. The outputs are the distance from the hand, the distance from the object, and the part label of the hand. A functional unit that processes information from the head camera 3 converts the output of the decoder g405 into information indicating the initial shape and posture of each finger, information indicating the label of each finger, a point cloud of each finger, and information indicating the initial position of the object to be operated. The configuration example shown in FIG. 9 is just an example, and the present invention is not limited to this.

[0067] FIG. 10 is a diagram showing an example of the processing contents and the processing procedure of the control device according to the present embodiment. In this example, an object to be operated is also placed on the operator's side. In this example, the operator actually operates this object. In addition, the reference symbols g502, g503, and g504 correspond to, for example, the hand tip posture estimation unit 43, the manipulation object posture estimation unit 44, and the contact estimation unit 45. In addition, the reference symbols g503 and g504 are formed of, for example, a network such as a trained CNN (Convolutional Neural Network).

[0068] The process of reference numeral g502 is a first hand process, which normalizes the ROI (Region of Interest) image of the finger.

[0069] The processing of reference symbol g503 receives as input information indicating the initial shape and posture of each finger converted by a functional unit that processes information from the head camera 3 in Fig. 9, information indicating the label of each finger, the point cloud of each finger, information indicating the initial position of the operation object, information indicating the camera pose, and the output of the processing of reference symbol g502. The processing of reference symbol g503 is a second hand process that predicts the finger joint positions for each palm camera, and estimates the posture of the operation object.

[0070] The process of reference symbol g504 is a third hand process that uses vertex information from the mesh to generate a point cloud using a MANO (hand Model with Articulated and Non-rigid defOrmations) (see, for example, Reference 1) network, and adjusts the estimated pose of the operating object.

[0071] The g505 process is a fourth hand process that generates a mesh using a Signed Distance Function (SDF) (see, for example, Reference 2) decoder network that learns a scalar field, which is a 3D representation. The g505 process also performs optimization by first prioritizing the cost of contour deviation, minimizing mesh penetration, and minimizing contact induction cost.

[0072] Reference 1: MANO ~ A pytorch Implementation of MANO hand model ~, Internet search 2023.10.23, https: / / github.com / otaheri / MANO Reference 2; Jeong Joon Park, Peter Florence, et al,. “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation”, 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

[0073] An overview of each process and an overview of the process procedure will be described with reference to FIG. Reference symbol g501 denotes an input, which is a plurality of frames of captured images. The captured plurality of frames of images are input to the hand tip posture estimation unit 43, the manipulation object posture estimation unit 44, and the contact estimation unit 45 (step S1).

[0074] (Step S2) (symbol g502) The hand posture estimation unit 43 detects the object and the hand from the image using, for example, a Kalman filter. Note that the two squares of symbol g5021 represent the object and the hand.

[0075] (Step S3) (reference numeral 5031) The hand posture estimation unit 43 uses the image captured by the head camera 3 to estimate the posture of the object.

[0076] (Step S4) (symbol g5041) The operation object posture estimation unit 44 uses the image captured by the palm camera 2 to adjust the object posture estimated in step S2.

[0077] (Step S5) (symbol g5032) The hand posture estimation unit 43 encodes the result of the first hand processing in which the image captured by the palm camera 2 is processed, and estimates the joint positions of each finger (second hand processing).

[0078] (Step S6) (symbol g5042) The hand posture estimation unit 43 optimizes the resolution including the mesh. More specifically, the hand posture estimation unit 43 uses vertex information from the mesh to generate a point cloud using a MANO (network (third hand processing).

[0079] (Step S7) (Code g505) The contact estimation unit 45 decodes the processing results of codes g5041 and g5043. More specifically, the contact estimation unit 45 generates a mesh using an SDF decoder network (fourth hand process). In addition, the contact estimation unit 45 performs optimization by prioritizing the cost of contour deviation, minimizing the mesh penetration amount, and minimizing the contact induction cost.

[0080] (Step S8) (Symbol g506) The contact estimation unit 45 outputs the estimation result. The output is, for example, the posture of the object to be operated, polygon data of the object to be operated, the posture of the hand, polygon data of the hand, the posture of the wrist, the contact point, and a normal indicating the direction in which the force acts at the contact point.

[0081] FIG. 11 is a diagram showing a schematic configuration of the process shown in FIG. 10 using a network. Images g601 to g605 captured by palm camera 2 are input to model g610 configured as a network.

[0082] The images captured by palm-first camera 21 to palm-third camera 23 are input to hand encoder g611. Hand encoder g611 convolves which pixels in the images represent which fingers, and estimates that the image is of a finger. Next, a 3D (three-dimensional) Convolutional Neural Network (CNN) g612 estimates the joints and fingers in the 2D images. The output includes, for example, a 3D heatmap and a softmax function (g613). The decoder of the code g614 converts from 2D to 3D. The output of the decoder g614, information indicating the initial position of the hand, and information indicating the initial posture of the fingers (symbol g615) are input to the hand decoder g616 (Mono Model Fitting). As a result, the hand decoder g616 outputs vertices g617 of the hand (including the fingers).

[0083] Moreover, the images captured by the fourth palm camera 24 to the fifth palm camera 25 are input to a manipulation object encoder g421. Then, the object encoder g621 estimates the position of the manipulation object and labels the manipulation object. The encoder g622 reconstructs the three-dimensional position from the labeled data. Note that during reconstruction, constraints are imposed and optimization is performed to prevent the contact point between the operation object and the hand from sinking in. The output g623 is the result of such optimization.

[0084] (Mounting example) Next, we will explain an example of a prototype mount and an example of wearing the mount on the hand. FIG. 12 shows an example of a prototype mount and an example of the mount being worn on the hand. Image g651 is an example of a prototype mount. The prototype mount is for right-handed people. Image g652 shows the prototype mount attached to the right hand. Image g653 is an example of picking up an object with precision grip. In this example, the mount of this embodiment does not interfere with gripping. Image g654 is an example of the orientation of each palm camera 21-25 when the manipulation object is precisely grasped. In this example, there is no problem with the orientation of the palm fourth camera and palm fifth camera for photographing the manipulation object and the orientation of the other palm cameras (palm first camera 21-palm third camera 23). Image g655 is an example of gripping the operation object with a strong grip. In this example, palm-first camera 21 and palm-third camera 23 can capture images of the operator's fingers so that they can be observed.

[0085] The shape, wearing example, and gripping example of the mount shown in Fig. 12 are merely examples and are not limited to these. For example, a mount for the left hand may be created and worn on the left hand, or separate mounts may be created for the left and right hands and worn on each hand.

[0086] (Another example of camera position for palm-mounted camera) An example of the arrangement of the cameras in palm camera 2 has been described with reference to FIG. 1, but the arrangement in FIG. 1 is just one example, and the following arrangement may also be used. 13 is a diagram showing another example of the camera position in the palm camera that places importance on the fingertip contour matching process. This example is an example of a layout that allows a desired finger image to be captured with emphasis on the fingertip contour matching process.

[0087] Reference numeral g700 denotes a first arrangement example. The position of palm third camera 23 may be within a range of area g701 enclosed by a square. Reference symbol g710 is a second example of placement. The position of palm-third camera 23 may be within the range of area g713 enclosed by a square. Arrows g711 and g712 each indicate a length and position where the camera does not hit the wrist even if the wrist is shook from side to side, for example, as in the gesture of saying "bye-bye." Triangle g714 indicates an example of the shooting range of palm-third camera 23.

[0088] In the first and second arrangements, for example, when the camera is placed on the palm, the fingers can be measured from closer. However, depending on the curvature of the palm, the camera and the mount may be pushed by the palm and move, requiring recalibration. In the first and second arrangement examples, for example, when the finger is placed on the side of the palm, it is possible to measure the finger regardless of the curvature of the palm. However, when the finger is placed on the side of the palm, it is necessary to place the finger at a position long enough that it does not come into contact with the arm even when the wrist is moved.

[0089] Symbol g720 is a third example of arrangement. The position of palm-third camera 23 may be within the range of area g721 enclosed by a rectangle. In this way, palm-third camera 23 may be located near the wrist. Also, as long as the position of palm-third camera 23 is within area g721, it may be at the left or right end. Triangle g722 represents an example of the imaging range when palm-third camera 23 is at the left end of the wrist. Triangle g723 represents an example of the imaging range when palm-third camera 23 is at the center of the wrist.

[0090] In the case of the wrist, the camera should be placed in a position and posture that makes it easy to observe the shape of the thumb. However, in this case, depending on the angle of the wrist, the fingers may not be visible (not within the shooting range), so it is preferable to add a camera on the palm or on the side of the palm.

[0091] 13 is merely an example, and the present invention is not limited to this example. For example, the arrangement may be one that places importance on distance measurement, as shown in FIG. FIG. 14 is a diagram showing another example of the camera position of the palm camera that places importance on distance measurement. The symbol g800 is an example of distance measurement using a stereo camera. The circles g801 and g802 are cameras that receive light. The square g803 represents the camera system as an area. In this case, it is possible to generate a dense disparity image using learning stereo.

[0092] The example of code g810 is an example in which the stereo camera of code g800 is replaced with a small camera that can also obtain depth information. Circle g811 is a light source. Circle g812 is a camera. Square g813 represents the camera system as an area.

[0093] The example of code g820 is an example in which a camera that can also obtain depth information of code g810 is moved from the palm to the palm surface. Circle g821 is a light source. Circle g822 is a camera. Square g823 represents the camera system as an area.

[0094] The example of code g830 is an example of moving a camera to the wrist, which can also obtain depth information of code g810. Circle g831 is a light source. Circle g832 is a camera.

[0095] The example of code g840 is an example in which the camera on the palm side of code g810 is replaced with a camera that can also obtain depth information. Circles g841 and g842 correspond to the palm first camera 21. Circle g841 is a light source. Circle g842 is a camera. Circles g842 and g843 correspond to the palm second camera 22. Circle g843 is a light source. Circle g844 is a camera.

[0096] The example of code g850 is an example in which the number of cameras is reduced to also obtain depth information of the wrist in code g850.

[0097] It should be noted that the example shown in FIG. 14 is merely an example, and the arrangement, cameras, and number of cameras for distance measurement are not limited to this example.

[0098] As described above, in this embodiment, the palm camera is attached at a position and orientation that allows the side and inside of the fingers to be continuously observed even if they are hidden by self-occlusion when viewed from a camera attached around the head or chest. This palm camera estimates the pose and polygons of the object and hand with high accuracy, and performs learning that can estimate the contact points and normals between the object and the hand from the estimated pose and polygons.

[0099] As a result, according to this embodiment, it is possible to measure how a person touches an object with high mobility and precision (the exact contact points and normals between a finger and an object, and between fingers).

[0100] <Second embodiment> In the first embodiment, a method for estimating the shape and posture of the hands of an operator remotely operating the robot 5 was described, but it is also possible to have the robot learn the movement and posture of the hands when actually grasping and operating an object while wearing the palm camera 2 and head camera 3.

[0101] Fig. 15 is a diagram showing an example of the configuration of a human hand posture estimating system according to this embodiment. As shown in Fig. 15, a human hand posture estimating system 1A includes, for example, a palm camera 2, a head camera 3, and a control device 4A. The control device 4A includes, for example, a first acquisition unit 41, a second acquisition unit 42, a hand posture estimation unit 43, an operating object posture estimation unit 44, a contact estimation unit 45, a control command generation unit 46, an output unit 47, a third acquisition unit 48, and a learning unit 40.

[0102] When an operator wears palm camera 2 and head camera 3 and performs an operation on an object (e.g., gripping, grabbing, lifting, etc.), learning unit 40 acquires image data captured by palm camera 2 and head camera 3 via first acquisition unit 41 and second acquisition unit 42. Learning unit 40 uses the acquired image data as teacher data to train hand posture estimation unit 43, manipulation object posture estimation unit 44, and contact estimation unit 45.

[0103] FIG. 16 is a diagram showing an example of input and output to a model during learning. As shown in FIG. 16, image data captured by palm camera 2 and image data captured by head camera 3 are input to model g800. Model g800 outputs, for example, information indicating the posture of an object, polygon data of the object, information indicating the posture of the hand, polygon data of the hand, information indicating the posture of the wrist, and information indicating the contact points and normals between the hand and the object.

[0104] The learning unit 40 uses, for example, image data captured by the palm camera 2 and image data captured by the head camera 3 as training data to learn the model g800.

[0105] 16 is merely an example and is not limited thereto. For example, the input may be only image data captured by palm camera 2. In this case, learning unit 40 may use, for example, the image data captured by palm camera 2 as teacher data to learn model g800.

[0106] As described above, in this embodiment, the palm camera 2 described in the first embodiment is worn by an operator and a model is learned when the operator operates an object.

[0107] As a result, in this embodiment, the occlusion of fingers or objects in the captured image can be reduced when learning a model, so that the model can be learned with high accuracy. That is, according to this embodiment, it can be applied to a visual life log of where people touch the environment or objects, so that a human contact database can be created. Also, according to this embodiment, it can be used as a learning database when a robot imitates and learns human contact information. As a result, according to this embodiment, using the model trained in this manner, it is possible to accurately estimate information indicating the posture of an object, polygon data of the object, information indicating the posture of the hand, polygon data of the hand, information indicating the posture of the wrist, and information indicating the contact points and normals between the hand and an object.

[0108] In each of the above-described embodiments, the mount 27 having the palm camera 2 may be attached to at least one hand of the operator, and may be attached to both hands.

[0109] In addition, a program for realizing all or part of the functions of the control device 4 (or 4A) in the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the control device 4. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to storage devices that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0110] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0111] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0112] 1, 1A... Hand posture estimation system, 2... Palm camera, 3... Head camera, 4, 4A... Control device, 5... Robot, 6... Environmental sensor, 7... HMD, 21... Palm first camera, 22... Palm second camera, 23... Palm third camera, 24... Palm fourth camera, 25... Palm fifth camera, 26... Communication unit, 31... Camera, 32... Communication unit, 41... First acquisition unit, 42... Second acquisition unit, 43... Hand posture estimation unit, 44... Manipulation object posture estimation unit, 45... Contact estimation unit, 46... Control command generation unit, 47... Output unit, 48... Third acquisition unit, 49... Image generation unit, 40... Learning unit, 51... End effector, 52... Arm, 53... Drive unit, 54... Sensor, 55... Acquisition unit, 56... Output unit, 61... Camera, 62... Communication unit

Claims

1. An apparatus for estimating a hand posture of an operator using a captured image of the operator's hand, comprising: A hand image capturing unit that captures an image of the shape of the operator's finger from the pad side of the finger; a hand posture estimation unit that analyzes the acquired image and estimates the posture of the hand of the operator; A hand posture estimation device comprising:

2. a control command generation unit that generates an operation command value for a robot based on the estimated hand posture of the operator; The hand posture estimating device according to claim 1 , further comprising:

3. an operation object photographing unit that photographs an image of an operation object that is operated by the operator's hand; an operation object posture estimation unit that estimates a posture of the operation object from an image of the operation object; a contact estimation unit that estimates a contact point and a normal between the operation object and the hand from the estimated operation object posture and the hand posture; The hand posture estimating device according to claim 1 , further comprising:

4. The hand and finger photographing unit includes: The camera is installed in a direction to photograph the palm side of the operator's finger. The hand posture estimating device according to claim 1 .

5. The hand and finger photographing unit includes a plurality of photographing units, The first finger photographing unit is installed in a direction to photograph the inside of the finger from the palm side of the index finger of the operator, The second finger photographing unit is installed in a direction to photograph the inside of the fingers from the palm side on the little finger side of the operator. The hand posture estimating device according to claim 1 .

6. The hand and finger photographing unit includes: The device is installed in a direction in which the thumb of the operator is photographed from the vicinity of the little finger ball. The hand posture estimating device according to claim 1 .

7. the manipulation object photographing unit is a parallax image photographing camera that generates a parallax image, The hand posture estimation unit An image captured by the manipulation object capturing unit in a direction other than the back of the hand from the palm; The posture of the hand is estimated from the image captured by the hand and finger capture unit. The hand posture estimating device according to claim 3 .

8. a photographing unit that captures an image of the shape of the operator's hand from an upper part of the operator's palm, The hand posture estimation unit analyzes the images acquired by the imaging unit and the hand and finger imaging unit, and estimates the posture of the operator's hand. The hand posture estimating device according to claim 1 .

9. A method for estimating a hand posture of an operator using a captured image of the operator's hand, comprising: a hand and finger photographing unit obtains an image of the shape of the operator's finger from the pad side of the finger; a hand posture estimation unit that analyzes the acquired image and estimates the posture of the hand of the operator; Hand and finger shape estimation method.

10. A computer of a device for estimating a hand posture of an operator using a captured image of the operator's hand, acquiring an image of a shape of the operator's finger from a pad side of the finger; a hand posture estimation unit that analyzes the acquired image and estimates the posture of the hand of the operator; program.

Images