Remote control system, remote control device, remote control method, and program

Abstract

A remote control system for a robot having a manipulator comprises: a first imaging device for imaging a hand of an operator; a position detection unit for detecting the position of a specific portion of the hand of the operator from an image captured by the first imaging device; a position determination unit for determining a target position of the manipulator on the basis of the detected position of the specific portion; and a control unit for controlling the manipulator on the basis of the determined target position.

Description

Remote control system, remote control device, remote control method, and program

[0001] The present invention relates to a remote control system, a remote control device, a remote control method, and a program.

[0002] Technologies for remotely controlling robots and the like are being developed. In such systems, an operator remotely controls the robot by operating a controller while viewing an image displayed on 3D glasses or by operating the controller while wearing a data glove (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2019-215769

[0004] However, conventional technologies require controllers and data gloves, which makes it difficult for operators to operate them intuitively.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a remote control system, a remote control device, a remote control method, and a program that can be operated intuitively by an operator.

[0006] (1) In order to achieve the above-mentioned object, a remote control system according to one aspect of the present invention is a remote control system for a robot having a manipulator, and is equipped with a first image capturing device that captures an image of an operator's hand, a position detection unit that detects the position of a specific part of the operator's hand from the image captured by the first image capturing device, a position determination unit that determines a target position of the manipulator based on the detected position of the specific part, and a control unit that controls the manipulator based on the determined target position.

[0007] (2) In one aspect of the remote control system according to (1) above, the position detection unit may include a segmentation unit that extracts the specific part from the image captured by the first image capture device, a skeleton recognition unit that recognizes a three-dimensional skeleton of the operator's hand from the image of the extracted specific part, and a skeleton coordinate calculation unit that calculates the coordinates of the positions of the operator's fingertips and wrist in the three-dimensional skeleton in a coordinate system of the operator's space (the space in which the operator exists) and calculates the orientation of the operator's hand.

[0008] (3) In a remote control system according to one aspect of (1) or (2) above, the remote control system may include a second camera capable of photographing the manipulator, and a presentation image generating unit that generates a presentation image to be presented to the operator by combining an image of the manipulator photographed by the second camera with an image of the manipulator photographed by the second camera by offsetting the coordinates of the operator's hand photographed by the first camera.

[0009] (4) In one aspect of the remote control system according to (3) above, the remote control system may include a sensor that detects the joint angles of the manipulator, a hand coordinate calculation unit that calculates the coordinates of the fingertip positions of each finger of the manipulator and the coordinates of the wrist using an image of the manipulator captured by the second image capture device and the joint angles detected by the sensor, and a coordinate conversion unit that converts the coordinates of the fingertip and the coordinates of the wrist of the operator detected by the position detection unit by moving and rotating them based on offset values, and the presentation image generation unit may generate the presentation image using the converted coordinates.

[0010] (5) In a remote control system according to one aspect of (3) or (4) above, the first image capturing device may be a stereo camera or may be composed of three or more cameras whose image capturing ranges at least partially overlap, and the second image capturing device may be a stereo camera, composed of three or more cameras, or may be composed of three or more cameras whose image capturing ranges at least partially overlap, and the distance between the operator's shoulder and the first image capturing device may be equal to the distance between the base of the manipulator and the position of the second image capturing device.

[0011] (6) In the remote operation system according to one aspect of (4) above, the coordinate conversion unit may convert the coordinates of the operator's fingertips and the coordinates of the wrist by moving and rotating them based on the coordinates after offsetting the wrist of the hand held by the manipulator, output the orientation of the operator's hand after the coordinate conversion, the position coordinates of the operator's fingertips after the coordinate conversion, and the offset value to be offset after the coordinate conversion to the presentation image generation unit, and output the position coordinates of the operator's fingertips after the coordinate conversion and the offset value after the coordinate conversion to the position determination unit.

[0012] (7) In order to achieve the above-mentioned object, a remote control device according to one aspect of the present invention is a remote control device for remotely controlling a robot having a manipulator and controlling the manipulator, and is equipped with a position detection unit that detects the position of a specific part of the operator's hand from an image captured by a first imaging device that images the operator's hand, a position determination unit that determines a target position of the manipulator based on the detected position of the specific part, and an output unit that outputs the determined target position to the robot.

[0013] (8) In order to achieve the above-mentioned object, a remote control method according to one aspect of the present invention is a remote control method for remotely controlling a robot having a manipulator and controlling the manipulator, in which a position detection unit detects the position of a specific part of the operator's hand from an image captured by a first imaging device that captures an image of the operator's hand, a position determination unit determines a target position of the manipulator based on the detected position of the specific part, and an output unit outputs the determined target position to the robot.

[0014] (9) In order to achieve the above object, one aspect of the present invention provides a program that causes a computer of a remote control device that remotely controls a robot having a manipulator and controlling the manipulator to detect the position of a specific part of the operator's hand from an image captured by a first image capture device that captures an image of the operator's hand, determines a target position of the manipulator based on the detected position of the specific part, and outputs the determined target position to the robot.

[0015] According to the above (1) to (9), the operator can intuitively operate the robot without wearing a data glove or the like.

[0016] 1 is a diagram illustrating an example of a schematic configuration of a remote control system according to an embodiment; FIG. 2 is a diagram illustrating an example of an image of an operator's hand and a position at which it is detected according to an embodiment; FIG. 3 is a diagram illustrating an example of an image of a robot's hand according to an embodiment; FIG. 4 is a diagram illustrating correspondence between an operator's hand and a robot's hand according to an embodiment, and an example image presented on goggles; FIG. 5 is a diagram illustrating an example of a configuration of a remote control system according to an embodiment; FIG. 6 is a diagram illustrating example data processed by a remote control device according to an embodiment; FIG. 7 is a diagram illustrating example data processed by a remote control device according to an embodiment; FIG. 8 is a diagram illustrating example data processed by a remote control device according to an embodiment, and an example image generated by the remote control device according to an embodiment; FIG. 9 is a diagram illustrating a coordinate system of an operator's space; FIG. 10 is a diagram illustrating a coordinate system of a robot's space; FIG. 11 is a diagram for explaining a coordinate system of a first sensor and a detection value of a second sensor; FIG. 12 is a flowchart of processing of a remote control system according to an embodiment;

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component has been appropriately changed so that each component can be recognized. In all drawings used to explain the embodiments, components having the same function are designated by the same reference numerals, and repeated explanations will be omitted. In addition, "based on XX" in this application means "based on at least XX" and includes cases where the component is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where the component is based on XX after calculation or processing. "XX" is any element (for example, any information).

[0018] 1 is a diagram showing an example of the schematic configuration of a remote control system according to this embodiment. The remote control system 1 includes, for example, a robot 2, a remote control device 4, goggles 5, and a first image capturing device 6. The robot 2 includes, for example, a second image capturing device 21, an arm 24 (manipulator), and a hand 25 (manipulator). The robot 2 also includes a body 30, wheels 31, a head 32, a power source, a communication unit, sensors, actuators, etc.

[0019] The robot 2 moves on wheels 31. The body 30 has, for example, one end of an arm 24 attached via a joint on each side thereof, and a cylindrical support extending upward from the top surface thereof. A head 32 is provided at the end of the support. The second image capturing device 21 is attached, for example, to the head 32. The hand 25 is attached to the other end of the arm 24 via a joint. The hand 25 is, for example, a multi-fingered hand having three or more fingers. The second image capturing device 21 captures an image of an area including the hand 25. The second image capturing device 21 is, for example, a stereo camera. The second image capturing device may also be composed of three or more cameras, or may be composed of three or more cameras whose capturing ranges at least partially overlap.

[0020] The goggles 5 display the image output by the remote control device 4. The operator remotely controls the robot 2 while viewing the image displayed on the goggles 5, without using a conventional controller or gloves. The first image capturing device 6 is attached to the goggles 5, for example. The first image capturing device 6 captures an image of an area including the operator's hands. The first image capturing device 6 is, for example, a stereo camera.

[0021] The remote control device 4 associates the operator's hand with the hand 25 of the robot 2 using an image of the area including the operator's hand captured by the first image capturing device 6 and an image of the area including the hand 25 captured by the second image capturing device 21. Based on the association result, the remote control device 4 generates a presentation image to be presented on the goggles 5. The remote control device 4 controls the movement of the arm 24 and hand 25 of the robot 2 using the movement of the operator's hand as an operation input.

[0022] [Photographing the operator's hand, photographing the robot's hand, and associating them] Next, photographing the operator's hand, photographing the robot's hand, and associating them will be described. In the following example, it is assumed that the hand 25 of the robot 2 has three fingers, each of which corresponds to a human thumb, index finger, and ring finger.

[0023] 2 is a diagram showing an example of capturing an image of an operator's hand and detecting the position according to this embodiment. As shown in FIG. 2, the first image capturing device 6 captures an image of an area including the operator's fingertips, palm, and wrist. Reference symbol g15 indicates an example of the capturing range of the first first image capturing device 6, and reference symbol g16 indicates an example of the capturing range of the second first image capturing device 6. Reference symbol g10 indicates the operator's arm, reference symbol g11 indicates the position of the tip of the thumb, reference symbol g12 indicates the position of the tip of the index finger, reference symbol g13 indicates the position of the tip of the ring finger, and reference symbol g14 indicates the position of the wrist.

[0024] The remote control device 4 uses the captured image to detect the position of the specific part, which is the position g11 of the operator's thumb, the position g12 of the operator's index finger, the position g13 of the operator's ring finger, and the position g14 of the operator's wrist. The detection method will be described later.

[0025] 3 is a diagram showing an example of capturing an image of a robot hand according to this embodiment. As shown in FIG. 3, the second camera 21 captures an image of an area including the fingertips, palm, and joints corresponding to the wrist of the robot hand. Reference symbol g21 indicates an example of the capturing range of the first second camera 21, and reference symbol g22 indicates an example of the capturing range of the second second camera 21. The image indicated by reference symbol g30 is an example of an image captured by the second camera 21. Reference symbol g31 indicates the position of the tip of the thumb, reference symbol g32 indicates the position of the tip of the index finger, reference symbol g33 indicates the position of the tip of the ring finger, and reference symbol g34 indicates the position of the wrist.

[0026] Using the captured image, the remote control device 4 detects, as the positions of the specific parts, the position g31 of the thumb tip of the hand 25, the position g32 of the index finger tip of the hand 25, the position g33 of the ring finger tip of the hand 25, and the position g34 of the wrist of the hand 25. The detection method will be described later.

[0027] 4 shows an example of the correspondence between the operator's hand and the robot's hand, and an example of an image displayed on the goggles in this embodiment. For example, in FIG. 4, an image is displayed on the goggles 5 excluding the star marks indicating the positions g21 to g24 of the operator's fingertips and wrist and the star marks indicating the positions g31 to g34 of the fingertips and wrist of the hand 25 of the robot 2. The image g100 of the operator's hand is a hand image g100 generated by the remote control device 4, and the image g110 is, for example, an image in which the outline of the hand 25 is a line drawing. In this embodiment, for example, the hand image g110 captured by the second image capturing device 21 is offset by a predetermined distance L from the image g100 of the operator's hand, and is displayed on the goggles 5.

[0028] The remote control device 4 then links the fingertip position g21 of the operator's thumb with the fingertip position g31 of the thumb of the hand 25, links the fingertip position g22 of the operator's index finger with the fingertip position g32 of the index finger of the hand 25, links the fingertip position g23 of the operator's ring finger with the fingertip position g33 of the ring finger of the hand 25, and links the wrist position g24 of the operator with the wrist position g34 of the hand 25.

[0029] 5 is a diagram showing an example of the configuration of a remote control system according to this embodiment. The remote control system 1 includes, for example, a robot 2, a remote control device 4, goggles 5, and a first image capturing device 6.

[0030] The robot 2 and the remote control device 4 are connected to each other via a wired network or a wireless network. The goggles 5 and the remote control device 4 are connected to each other via a wired network or a wireless network. The first image capturing device 6 and the remote control device 4 are connected to each other via a wired network or a wireless network.

[0031] The robot 2 includes, for example, a second imaging device 21, a first sensor 22 (sensor), a second sensor 23 (sensor), an arm 24 (manipulator), a hand 25 (manipulator), an actuator 26, and a control unit 27. As described above, the robot 2 includes a body 30, wheels 31, a head 32, a power source, a communication unit, and the like.

[0032] The remote control device 4 includes, for example, a segmentation unit 401 (position detection unit), a skeleton recognition unit 402 (position detection unit), a skeleton coordinate calculation unit 403 (position detection unit), a model generation unit 404, a hand coordinate calculation unit 405, an offset unit 406, a calculation unit 407, a coordinate conversion unit 408, a hand image generation unit 409 (presentation image generation unit), a presentation image generation unit 410, a first output unit 411, a calculation unit 412 (position determination unit), a first difference calculation unit 413 (position determination unit), a first integration unit 414 (position determination unit), a calculation unit 415 (position determination unit), a second difference calculation unit 416 (position determination unit), a second integration unit 417 (position determination unit), an initial position memory unit 418 (position determination unit), a calculation unit 419 (position determination unit), and a second output unit 420. The remote control device 4 also includes, for example, a power supply, a communication unit, a storage unit, and the like.

[0033] (Goggles) The goggles 5 are, for example, virtual reality (VR) goggles, a head-mounted display (HMD), etc. The goggles 5 include, for example, a display unit, a communication unit, a power supply, etc.

[0034] (First Image Capturing Device) The first image capturing device 6 is, for example, a stereo camera, and is attached to, for example, the goggles 5. The first image capturing device 6 also includes, for example, a power source, a communication unit, and the like.

[0035] (Robot) The robot 2 is an example of a moving body. The robot 2 includes at least one pair of an arm 24 and a hand 25.

[0036] The second image capturing device 21 is, for example, a stereo camera, and is attached to the head 32, for example.

[0037] The first sensor 22 is, for example, an encoder, and is a sensor that detects the angle of each joint of the hand 25. The coordinate system of the detection value detected by the first sensor 22 is the robot coordinate system.

[0038] The second sensor 23 is, for example, an encoder, and is a sensor that detects the angle of each joint of the arm 24. The coordinate system of the detection value detected by the second sensor 23 is the robot coordinate system.

[0039] The arm 24 has a joint, and the joint is provided with a second sensor 23 and an actuator 26 .

[0040] The hand 25 has a joint, and the joint is provided with a first sensor 22 and an actuator 26 .

[0041] The actuators 26 are attached to the joints of the arm 24 and the hand 25 .

[0042] The control unit 27 acquires the fingertip position target values ​​and wrist position target values ​​output by the remote control device 4, and controls the operation of the arm 24 and the hand 25 based on the acquired fingertip position target values ​​and wrist position target values, for example, on inverse kinematics.

[0043] (Remote Control Device) The remote control device 4 generates an image to be presented on the goggles 5 using images captured by the first image capturing device 6 and the second image capturing device 21. The remote control device 4 generates a fingertip position target value and a wrist position target value using the images captured by the first image capturing device 6 and the second image capturing device 21. Each unit of the remote control device 4 repeats processing at predetermined time intervals.

[0044] The segmentation unit 401 acquires, for example, a stereo image of a region including the operator's hand and arm captured by the first image capturing device 6. The segmentation unit 401 is, for example, a trained deep neural network (DNN), which receives an image as input and extracts a hand region (a region including fingers, hands, wrists, etc.) from the image. The segmentation unit 401 receives, for example, stereo images and training data (hand regions) for multiple people and learns to output the hand regions.

[0045] The skeleton recognition unit 402 is, for example, a trained DNN, and recognizes the three-dimensional skeleton of the operator's hand (fingers, joints, wrist) and arm by inputting an image of the extracted hand region. Note that the skeleton recognition unit 402 is trained to input, for example, images of the hand region and correct answer data (the three-dimensional skeleton of the operator's hand (fingers, joints, wrist) and arm) for multiple people, and output the three-dimensional skeleton of the operator's hand (fingers, joints, wrist) and arm.

[0046] The skeleton coordinate calculation unit 403 calculates the coordinates of the positions of the fingertips and the wrist (base of the hand) in the three-dimensional skeleton recognized by the skeleton recognition unit 402 in the camera coordinate system of the first image capture device 6 by a well-known method. The skeleton coordinate calculation unit 403 also calculates the orientation of the operator's hand by a well-known method. The skeleton coordinate calculation unit 403 calculates the calculated position coordinates Xu of the operator's fingertips. ｋ (k=1 to 3) and the position coordinate Xu of the operator's wrist ｗ and the operator's hand direction Θu ｗ is output to the coordinate conversion unit 408.

[0047] The model generation unit 404 is, for example, a trained DNN, and generates a mesh model including the hand and arm by inputting the image of the hand region output by the segmentation unit 401 to the trained DNN. Note that the model generation unit 404 inputs, for example, images of the hand region and correct answer data (mesh models) for multiple people, and trains to output the mesh models.

[0048] The hand coordinate calculation unit 405 acquires, for example, stereo images captured by the second image capture device 21. The hand coordinate calculation unit 405 acquires each joint angle of the hand 25 detected by the first sensor 22 and each joint angle of the arm 24 detected by the second sensor 23. Using the acquired images and joint angles, the hand coordinate calculation unit 405 calculates the coordinates of the fingertip positions of each finger of the hand 25 and the coordinate of the wrist (the base of the hand 25) in the camera coordinate system of the second image capture device 21 using a well-known method. Note that in addition to the camera recognition results of the spatial coordinates X-Yr-Zr positions of the arm 24 and hand 25, the hand coordinate calculation unit 405 recognizes each of the joint positions of the arm 24 and hand 25 using sensors and then recognizes the spatial coordinates X-Yr-Zr positions of the arm 24 and hand 25 by, for example, combining (e.g., weighted averaging) the positions.

[0049] The offset unit 406 sets an offset value of the coordinates of the wrist of the hand 25 relative to the coordinates of the wrist of the operator. The offset value may be stored in advance in the offset unit 406, or may be set or selected by the operator.

[0050] The calculation unit 407 calculates an offset value DXr by which the wrist position coordinates output by the offset unit 406 are offset to the wrist position coordinates of the hand 25 output by the hand coordinate calculation unit 405. ｗ＿ｏｆｔ The coordinates of the hand 25 are offset by adding the above to the coordinate Xr ｗ＿ｏｆｔ Output includes.

[0051] The coordinate conversion unit 408 acquires coordinates relating to the operator's hand output by the skeleton coordinate calculation unit 403. The coordinate conversion unit 408 acquires coordinates relating to the offset hand 25 output by the calculation unit 407. The coordinate conversion unit 408 converts the coordinates of the operator's fingertips and the coordinates of the wrist by moving and rotating them based on the coordinates of the wrist of the offset hand 25 in the camera coordinate system of the second image capture device 21. The coordinate conversion unit 408 calculates the direction Θu of the operator's hand after the coordinate conversion. ｗ and the position coordinates X'vk of the operator's fingertip after the coordinate transformation, and the offset Xr of the wrist after the coordinate transformation. ｗ＿ｏｆｔ is output to the hand image generation unit 409.

[0052] The hand image generation unit 409 generates an image of the operator's hand using the coordinates converted by the coordinate conversion unit 408 and the mesh model output by the model generation unit 404 .

[0053] The presentation image generation unit 410 generates a presentation image by combining the image of the hand 25 captured by the second image capturing device 21 with the image of the operator's hand generated by the hand image generation unit 409. The presentation image generation unit 410 generates a presentation image expressed in the camera coordinate system of the second image capturing device 21 (coordinate system in the robot's space).

[0054] The first output unit 411 outputs the presentation image generated by the presentation image generation unit 410 to the goggles 5 .

[0055] The calculation unit 412 converts the converted coordinates X'vk (k=1 to 3) of the operator's fingertip position output by the coordinate conversion unit 408 into the offset coordinates Xr of the wrist of the hand 25 output by the coordinate conversion unit 408. ｗ＿ｏｆｔ is subtracted at predetermined intervals, and the subtracted dXc ｋ to the first difference calculation unit 413.

[0056] The first difference calculation unit 413 calculates the dXc output from the calculation unit 412. ｋ Regarding the difference from the previous value, ΔdXc ｋ is calculated at predetermined time intervals.

[0057] The first integration unit 414 calculates the difference ΔdXc output by the first difference calculation unit 413. ｋ is integrated for a predetermined time.

[0058] The calculation unit 415 calculates the offset coordinate Xr of the wrist of the hand 25 output by the coordinate conversion unit 408 from the integration result output by the first integration unit 414. ｗ＿ｏｆｔ By adding ｃｍｄ＿ｋ As a result, although the joint structures and ranges of motion of the operator's hand and the arm 24 of the robot 2 are different, it is possible to operate the hand 25 while absorbing these differences.

[0059] The second difference calculation unit 416 calculates the coordinates Xu of the fingertip position of the operator output by the skeleton coordinate calculation unit 403. ｋ (k = 1 to 3), coordinate of wrist position Xu ｗ and hand direction Θr ｗ Regarding the difference from the previous value, Δdu ｋ is calculated at predetermined time intervals.

[0060] The second integration unit 417 calculates the difference Δdu output by the second difference calculation unit 416. ｋ is integrated for a predetermined time.

[0061] The initial position storage unit 418 stores the initial position of the hand 25. The initial position may be determined from an image captured by the second image capturing device 21, or may be a predetermined position.

[0062] The calculation unit 419 calculates the wrist position target value Xr by adding the initial position output by the initial position storage unit 418 to the integrated result output by the second integration unit 417. ｃｍｄ＿ｗ This makes it possible to operate the wrist of the robot 2 while absorbing the differences between the operator's hand and the arm 24 of the robot 2, even though the hand of the operator and the arm 24 of the robot 2 are in different spaces, the joint structure from the wrist to the hand is different, the range of motion is different, and the spatial position and posture (angle) are also different.

[0063] The second output unit 420 receives the fingertip position target value Xr ｃｍｄ＿ｋ and the wrist position target value Xr output by the calculation unit 419. ｃｍｄ＿ｗ is output to robot 2.

[0064] [Example of Processed Data] Next, an example of processed data of each part of the remote control device 4 will be described. Figures 6 and 7 are diagrams showing examples of processed data of the remote control device of this embodiment. Image g100 in Figure 6 is an example of an image of the operator's hand (including fingers, palm, wrist, and part of arm) captured by the first image capturing device 6.

[0065] 6 is an example of a hand region g111 extracted by the segmentation unit 401. In this embodiment, the hand region includes the fingertips, fingers, palm, wrist, and part of the arm.

[0066] 6 is an example of a skeleton image recognized by the skeleton recognition unit 402. A skeleton image like image g120 includes, for example, information representing the skeleton by connecting the fingertips, the joints of each finger, the base of the fingers, the wrist (base of the hand), and each joint and position with a line.

[0067] An image g130 in FIG. 6 is an example of an image of a mesh model generated by the model generation unit 404.

[0068] 7 is an image for explaining the coordinates calculated by the skeleton coordinate calculation unit 403. Points g141 to g143 correspond to the coordinates Xu of the operator's fingertip. ｋ (k=1 to 3), and the point g141 is the coordinate of the fingertip of the thumb Xu １ and the point g142 is the coordinate of the tip of the index finger Xu ２ and the point g143 is the fingertip coordinate Xu ３ The point g144 is the coordinate Xu of the wrist (base) of the operator. ｗ In addition, the direction of the operator's hand is Θu ｗ and the component in the x-axis direction is θux ｗ and the component in the y-axis direction is θuy ｗ and the z-axis component is θuz ｗ is.

[0069] 7 is an image for explaining the coordinates calculated by the coordinate conversion unit 408. Points g151 to g153 are obtained by dividing the wrist coordinates by Xr ｗ Move to, and turn your hand to Θr ｗ The point g151 is the coordinate X'vk (k=1 to 3) of the fingertip of the operator after transformation when the fingertip is moved and rotated to the position shown in FIG. １ The point g152 is the transformed coordinate Xv2 of the index fingertip, and the point g153 is the transformed coordinate Xv ３ The point g154 is the coordinate Xr of the wrist (base) of the operator after transformation. ｗ＿ｏｆｔ In addition, the direction of the operator's hand after conversion is Θu ｗ is.

[0070] 8A and 8B are diagrams showing examples of processed data and generated images of the remote control device of this embodiment. Image g200 is an example of an image of the hand 25 including a part of the arm 24 of the robot 2 captured by the second image capturing device 21.

[0071] The image g210 is an image for explaining the coordinates calculated by the hand coordinate calculation unit 405. The hand angle input from the first sensor 22 is φx ｋｊ、 φy ｋｊ , φz ｋｊ (k=1 to 3, j=1, 2). The arm angle input from the second sensor 23 is Θx ｉ、 Θy ｉ , Θz ｉ (i=1 to 4). Points g211 to g213 are the coordinates Xr of the fingertip positions of the hand 25. ｋ (k=1 to 3), and the point g211 is the coordinate Xr １ The point g212 is the coordinate Xr ２ The point g213 is the coordinate Xr ３ The point g214 is the position coordinate Xr of the wrist of the hand 25. ｗ The direction of the hand 25 is Θr ｗ and the component in the x-axis direction is θrx ｗ and the component in the y-axis direction is θry ｗ and the component in the z-axis direction is θrz ｗ is.

[0072] Image g220 is an example of a hand image generated by the hand image generation unit 409. The hand image is generated based on the mesh model generated by the model generation unit 404 and the coordinates converted by the coordinate conversion unit 408, and is an image of the virtual operator's hand. The virtual hand image includes star marks (coordinates) indicating the operator's fingertips, which are used to instruct control of the hand 25 of the robot 2. The virtual hand image may also include points indicating joints, lines connecting the joints, a star mark indicating the wrist position, and k=1 to 3 indicating the finger numbers. The display of each piece of information may be switched ON / OFF in response to an instruction from the operator.

[0073] Image g230 is an example of an image presented to the goggles 5 generated by the presentation image generation unit 410. The presentation image includes an image of a virtual operator's hand, an image of the hand 25 (including part of the arm 24) of the robot 2, and a work environment in which the robot 2 performs work. Furthermore, as in image g230, in the presentation image, the image of the virtual operator's hand is offset with respect to the position coordinates of the wrist of the hand 25 of the robot 2. The presentation image also includes star symbols (coordinates) indicating the operator's fingertips used to instruct control of the hand 25 of the robot 2. The presentation image may also include dots indicating the operator's joints, lines connecting the joints, a star symbol indicating the wrist position, and k = 1 to 3 indicating the finger numbers. The presentation image also includes star symbols (coordinates) indicating the fingertips of the hand 25 of the robot 2. The presentation image may also include dots indicating the joints of the hand 25, lines connecting the joints, a star symbol indicating the wrist position, and k = 1 to 3 indicating the finger numbers. The presentation image may also include dots indicating the joints of the hand 25, lines connecting the joints, a star symbol indicating the wrist position, and k = 1 to 3 indicating the finger numbers. The display of each piece of information may be switched on and off in response to an instruction from the operator. Furthermore, if the hand 25 has multiple fingers, the presented image may include correlation information indicating which finger of the operator is correlated with which finger of the hand 25 of the robot 2. The correlation information may correlate the operator's finger with the finger of the hand 25 of the robot 2, for example, by making the shape or color of the mark the same.

[0074] The processed image data shown in FIGS. 6 to 8 are merely examples and are illustrative diagrams for the purpose of explanation, and are not limited to these.

[0075] [Coordinate Systems] Next, each coordinate system will be described. Fig. 9 is a diagram showing a coordinate system in the operator's space. The origin O of the coordinate system in the operator's space is located at the center near the joints of the operator's left and right shoulders, with the up-down direction being the yu-axis direction, the left-right direction being the xu-axis direction, and the front-to-back direction being the zu-axis direction. Note that the coordinate system is the camera coordinate system of the first image capturing device 6.

[0076] 10 is a diagram showing a coordinate system in the robot's space. The origin O of the coordinate system in the robot's space is located at the center of the joint connecting the left and right arms 24 of the robot 2 to the body 30, and the up-down direction is the yr-axis direction, the left-right direction is the xr-axis direction, and the front-back direction is the zr-axis direction. The coordinate system is the camera coordinate system of the second image capturing device 21.

[0077] When capturing an image of the operator's hand, if the area of ​​the operator's hand is not within the image capturing range of the first image capturing device 6, the remote control device 4 may display, as a presented image, text or the like warning that the area is not within the image capturing range of the first image capturing device 6. Furthermore, when the operator starts remote operation, if the posture of the arm 24 and hand 25 of the robot 2 differs from the posture of the operator's hand, the remote control device 4 may present only images of the arm 24 and hand 25 of the robot 2 as presented images before the start of remote operation. Alternatively, the remote control device 4 may operate the arm 24 and hand 25 of the robot 2 to match the posture of the operator's hand at the start of remote operation.

[0078] [Coordinate system of the first sensor, measurement value of the second sensor] Fig. 11 is a diagram for explaining the coordinate system of the first sensor and the measurement value of the second sensor. As shown in Fig. 11, k = 1 to 3 is assigned to each finger of the hand 25. The angle of the hand detected by the first sensor 22 is expressed as φx ｋｊ、 φy ｋｊ , φz ｋｊ (k=1 to 3, j=1, 2). The subscript j is the number of the finger joint, with j=1 being the first joint and j=2 being the second joint. In this embodiment, each finger of the hand 25 of the robot 2 has two joints. The arm angle detected by the second sensor 23 is expressed as Θx as described above. ｉ、 Θyｉ , Θz ｉ (i=1 to 4). The subscript i is the number of the joint of the arm 24, and i=1 to 4 are assigned from the wrist to the shoulder.

[0079] In this embodiment, in addition to the camera recognition results of the spatial coordinate X-Yr-Zr positions of the arm 24 and hand 25 of the robot 2, the spatial coordinate X-Yr-Zr positions of the arm 24 and hand 25 are recognized using sensors to recognize the joint positions of the arm 24 and hand 25, and the spatial coordinate X-Yr-Zr positions of the arm 24 and hand 25 are recognized by, for example, synthesis (for example, weighted averaging).

[0080] [Processing Procedure] Next, a description will be given of an example of a processing procedure of the remote control system 1. Fig. 12 is a flowchart of processing of the remote control system according to this embodiment.

[0081] (Step S1) The first image capturing device 6 captures an image of the operator's hand (including the fingertips, palm, wrist, and part of the arm).

[0082] (Step S2) The second image capturing device 21 captures an image of the hand 25 of the robot 2 (including the fingertips, palm, wrist, and part of the arm 24).

[0083] (Step S3) The segmentation unit 401 inputs the image captured by the first image capturing device 6 into the trained DNN, thereby extracting a hand region (a region including fingers, hand, wrist, etc.) from the image.

[0084] (Step S4) The skeleton recognition unit 402 inputs the image of the extracted hand region into the trained DNN, thereby recognizing the three-dimensional skeleton of the operator's hand (fingers, joints, wrist) and arm.

[0085] (Step S5) The skeleton coordinate calculation unit 403 calculates, by a well-known method, the coordinates of the positions of the fingertips and the wrist in the coordinate system of the operator's space (the camera coordinate system of the first image capture device 6) in the three-dimensional skeleton recognized by the skeleton recognition unit 402. The skeleton coordinate calculation unit 403 calculates the orientation of the operator's hand by a well-known method.

[0086] The coordinates calculated by the skeleton coordinate calculation unit 403 are expressed as follows in the coordinate system of the operator's space shown in Fig. 9 : The coordinate Xuk of the operator's fingertip is expressed by the following formula (1). The coordinate Xuw of the operator's wrist is expressed by the following formula (2). The direction Θuw of the operator's hand is expressed by the following formula (3).

[0087] Xuk = [xuk, yuk, zuk] (k = 1 to 3)...Formula (1) Xuw = [xuw, yuw, zuw]...Formula (2) Θuw = [θux ｗ , θuy ｗ , θuz ｗ ]...Formula (3)

[0088] (Step S6) Using the acquired image and joint angles, the hand coordinate calculation unit 405 calculates the coordinates of the fingertip positions of each finger of the hand 25 and the coordinates of the wrist (the base of the hand 25) in the coordinate system of the robot's space (the camera coordinate system of the second image capture device 21) using a well-known method.

[0089] The coordinates calculated by the hand coordinate calculation unit 405 are expressed as follows in the coordinate system of the robot space shown in Fig. 10. The coordinate Xrk of the fingertip of the hand 25 of the robot 2 is expressed by the following equation (4). The coordinate Xrw of the wrist of the hand 25 is expressed by the following equation (5). The orientation Θrw of the hand 25 is expressed by the following equation (6).

[0090] Xrk=[xrk, yrk, zrk] (k=1 to 3)...Formula (4) Xrw=[xrw, yrw, zrw]...Formula (5) Θrw=[θrx ｗ , θry ｗ , θrz ｗ ]...Formula (6)

[0091] (Step S7) The calculation unit 407 calculates an offset value DXr by which the wrist position coordinate output by the offset unit 406 is offset to the wrist position coordinate of the hand 25 output by the hand coordinate calculation unit 405. ｗ＿ｏｆｔ Add.

[0092] (Step S8) The coordinate conversion unit 408 converts the coordinates of the operator's fingertip and wrist by moving and rotating them based on the coordinates of the wrist of the offset hand.

[0093] The coordinate transformation process of the coordinate transformation unit 408 will be described in detail below. First, the coordinate transformation unit 408 calculates the relative coordinate X'uk of the operator's fingertip with respect to the coordinate Xuw of the operator's wrist. The relative coordinate X'uk is expressed by the following equations (7) to (10).

[0094] X'uk=[x'uk,y'uk,z'uk](k=1-3)...Equation (7) x'uk=xuk-xuw...Equation (8) y'uk=yuk-yuw...Equation (9) z'uk=zuk-zuw...Equation (10)

[0095] Next, the coordinate transformation unit 408 calculates the coordinate X'vk of the operator's fingertip (the coordinate X'vk of the operator's fingertip after coordinate transformation) when the coordinate Xuw of the operator's wrist and the coordinate Xrw of the wrist of the hand 25 of the robot 2 are matched in the coordinate system of the robot space. The coordinate X'vk of the operator's fingertip after coordinate transformation is expressed by the following equations (11) to (14).

[0096] X'vk=[x'vk,y'vk,z'vk](k=1-3)...Equation (11) x'vk=x'uk+xrw...Equation (12) y'vk=y'uk+yrw...Equation (13) z'vk=z'uk+zrw...Equation (14)

[0097] Next, the coordinate conversion unit 408 calculates the coordinates of the operator's fingertips in the coordinate system of the robot's space (the coordinates of the operator's fingertips after rotation) when the image of the operator's hand (three-dimensional skeleton) is rotated so that the orientation of the operator's hand matches the orientation of the hand 25 of the robot 2. The angle Φ by which the image of the operator's hand (three-dimensional skeleton) is rotated is expressed by the following equations (15) to (18).

[0098] Φ=[φx, φy, φz]...Formula (15) φx=θux ｗ -θrx ｗ ...Formula (16) φy=θuy ｗ -θry ｗ ...Formula (17) φz=θuzｗ -θrz ｗ ...Formula (18)

[0099] In this case, the rotational coordinate transformation matrix Erot is expressed by the following equations (19) to (22).

[0100]

[0101] The coordinate transformation unit 408 uses this rotational coordinate transformation matrix Erot to calculate the coordinate X''vk of the operator's fingertip after rotation in the robot space. The coordinate X''vk of the operator's fingertip after rotation is calculated by the following equation (23).

[0102] X''vk Ｔ = [x''vk, y''vk, z''vk] Ｔ =E_rot(φx, φy, φz)X'uk Ｔ (k=1-3)...Formula (23)

[0103] Next, the coordinate conversion unit 408 offsets the image of the hand 25 of the robot 2 and the image of the operator's hand, thereby offsetting the coordinates of the operator's hand so that the operator can easily see the movements of both at the same time. ｗ＿ｏｆｔ is expressed by the following equation (24): ＊ vk is expressed by the following equations (25) to (28).

[0104] DXr ｗ＿ｏｆｔ = [DXr_x ｗ＿ｏｆｔ , DXr_y ｗ＿ｏｆｔ , DXr_z ｗ＿ｏｆｔ ]...Formula (24) X ＊ vk Ｔ = [x ＊ vk,y ＊ vk, z ＊ vk] (k=1 to 3)...Formula (25) x ＊ vk=x''vk+DXr_x ｗ＿ｏｆｔ ...Equation (26) y ＊ vk=y''vk+DXr_y ｗ＿ｏｆｔ ...Formula (27) z ＊ vk=z''vk+DXr_z ｗ＿ｏｆｔ ...Formula (28)

[0105] Furthermore, the coordinate conversion unit 408 calculates the absolute coordinate Xvk of the operator's hand in the coordinate system of the robot's space. The absolute coordinate Xvk is calculated using the following equations (29) to (32). Since the coordinate Xrw of the wrist of the hand 25 of the robot 2 is defined in the coordinate system of the robot's space, the coordinate Xrw of the operator's fingertip after offset is ＊ All that is required is to offset vk using the coordinate Xrw of the wrist of the hand 25 of the robot 2.

[0106] Xvk Ｔ = [xvk, yvk, zvk] (k=1 to 3)...Formula (29) xvk=x ＊ vk+xrw...Formula (30) yvk=y ＊ vk+yrw...Formula (31) zvk=z ＊ vk+zrw...Formula (32)

[0107] (Step S9) The model generation unit 404 inputs the image of the hand region output by the segmentation unit 401 into the trained DNN, and generates a mesh model including the hand and arm.

[0108] (Step S10) The hand image generation unit 409 generates an image of the operator's hand using the coordinates converted by the coordinate conversion unit 408 and the mesh model output by the model generation unit 404.

[0109] (Step S11) The presentation image generation unit 410 generates a presentation image by combining the image of the hand 25 captured by the second image capture device 21 with the image of the operator's hand generated by the hand image generation unit 409. If the size of the robot 2 differs from the size of the operator, the presentation image generation unit 410 adjusts the ratio between the size of the robot's hand 25 and the size of the virtual operator's hand image to match the size ratio between the robot 2 and the operator, and generates the presentation image.

[0110] (Step S12) The first output unit 411 outputs the presentation image generated by the presentation image generation unit 410 to the goggles 5.

[0111] (Step S13) The calculation unit 412, the first difference calculation unit 413, the first integration unit 414, and the calculation unit 415 calculate the fingertip position target value Xrｃｍｄ＿ｋ Ask for.

[0112] (Step S14) The second difference calculation unit 416, the second integration unit 417, and the calculation unit 419 calculate the wrist position target value Xr ｃｍｄ＿ｗ Ask for.

[0113] (Step S15) The second output unit 420 outputs the fingertip position target value Xr ｃｍｄ＿ｋ and the target wrist position value Xr ｃｍｄ＿ｗ is output to robot 2.

[0114] 12 are merely examples and are not limiting. For example, some processes may be performed simultaneously in parallel.

[0115] As described above, in this embodiment, the first imaging device 6 on the operator's side captures an image of the operator's hand, identifies the (three-dimensional) position of the imaged hand, and causes a remote robot arm to follow the position corresponding to the identified position.

[0116] As a result, according to this embodiment, an operation controller or data gloves are not required, and the operator can intuitively operate the system without wearing data gloves, etc. Furthermore, according to this embodiment, since an operation controller or data gloves are not required, the cost of the remote operation system 1 can be reduced compared to conventional systems.

[0117] [Positional Relationship Between the First and Second Cameras] Figure 13 is a diagram showing an example of the positional relationship between the first and second camera devices. As shown in Figure 13, the distance Lu between the operator's shoulder and the first camera device 6 on the operator's side and the distance Lr between the base of the arm 24 and the position of the second camera device 21 on the robot 2 side are set to be as close as possible. This allows the present embodiment to create a sense of unity between the body of the robot 2's arm 24 and the operator's arm (e.g., achieving ease of operation and spatial imagery). Note that if there is a large difference, the operator will perceive the apparent appearance due to image conversion. Therefore, it is better to have a small difference. Note that if the size of the robot 2 and the size of the operator are different, the distance between the shoulder and the camera device is set according to the size ratio between the robot 2 and the operator.

[0118] [Another Arrangement Example of the First and Second Cameras] Next, another arrangement example of the first and second camera devices will be described. FIG. 14 is a diagram showing another arrangement example of the second camera device. As shown in FIG. 14, the second camera device 21 may be a single camera. In this case, since the spatial coordinate X-Yr-Zr positions of the arm 24 and hand 25 cannot be recognized using the image of the second camera device 21, the image may be input to a trained DNN, for example, and depth estimation may be used in combination to recognize the spatial coordinate X-Yr-Zr positions of the arm 24 and hand 25. Alternatively, a distance measurement device may be provided in addition to the second camera device 21. Alternatively, the second camera device 21 may be an RGB (red, green, blue)-D camera that can also obtain depth information D.

[0119] 15 shows another example of the placement of the first image capturing device. It is desirable to use a stereo camera to improve the accuracy of recognizing the 3D image (3D segmentation) of the operator's arm and hand. Images g200 to g250 show another example of the placement of the stereo camera.

[0120] Image g200 shows an example in which the two first image capturing devices 6A-1 and 6A-2 are placed on the shoulders of an operator. In this case, the two first image capturing devices 6A-1 and 6A-2 may be placed near the left and right shoulders using, for example, a neckband g201.

[0121] Image g210 shows an example in which one first image capturing device 6-1 is placed on the goggles 5 and the other first image capturing device 6A-2 is placed, for example, on the operator's shoulder. In this case, for example, one first image capturing device 6A-2 may be placed near one shoulder using a neckband g211 or the like.

[0122] An image g220 is an example in which two first image capturing devices 6B-1 and 6B-2 are arranged around the operator using, for example, a stand.

[0123] Image g230 is an example in which one first image capturing device 6-1 is placed on the goggles 5, and another first image capturing device 6B-2 is placed around the operator using, for example, a stand.

[0124] Image g240 shows an example in which one first image capturing device 6B-1 is placed around the operator using, for example, a stand, and the other first image capturing device 6A-2 is placed, for example, on the operator's shoulder. In this case, for example, one first image capturing device 6A-2 may be placed near one shoulder using, for example, a neckband g211.

[0125] Image g250 is an example in which one first image capture device 6-1 is placed in the goggles 5. In this case, the captured image may be input to a trained DNN to perform depth estimation. Alternatively, a distance measurement device may be provided in addition to the first image capture device 6-1. Alternatively, the first image capture device 6-1 may be an RGB (red, green, blue)-D camera that can also obtain depth information D. This makes it possible to obtain 3D images (3D segmentation) of the operator's arms and hands even with a single camera.

[0126] 15, the first image capturing device 6 may be a stereo camera or may be configured with three or more cameras whose capturing ranges at least partially overlap each other. Note that the example of the arrangement of the first image capturing device 6 shown in FIG. 15 is just an example and is not limited to this.

[0127] The functions of the first difference calculation unit 413 and the first integration unit 414 described above may be realized by a model that integrates both functions. This model is, for example, a machine learning model such as a neural network. This machine learning model calculates the dXc output from the calculation unit 412. ｋ When input, the difference from the previous value ΔdXc ｋ The machine learning model is trained so that the integrated value (target fingertip relative position value) of Xr ｃｍｄ＿ｋ Furthermore, weighting may be applied to the high frequency components of the operator's correction operation and the amount of change.

[0128] The functions of the second difference calculation unit 416 and the second integration unit 417 described above may be realized by a model that integrates both functions. This model is, for example, a machine learning model such as a neural network. This machine learning model calculates the coordinates Xu of the operator's fingertip position output by the skeleton coordinate calculation unit 403. ｋ , the coordinate of the wrist position Xu ｗ and hand direction Θr ｗ When input, the difference Δdu from the previous value ｋ The machine learning model is trained to output the integral value (target relative wrist position value) of Xr ｃｍｄ＿ｗ Furthermore, weighting may be applied to the high frequency components of the operator's correction operation and the amount of change.

[0129] A program for implementing all or part of the functions of the remote control device 4 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the remote control device 4. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSI (Large Scale Integration) hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0130] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0131] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0132] REFERENCE SIGNS LIST 1... remote control system, 2... robot, 4... remote control device, 5... goggles, 6... first image capturing device, 21... second image capturing device, 22... first sensor, 23... second sensor, 24... arm, 25... hand, 26... actuator, 27... control unit, 30... body, 31... wheels, 32... head, 401... segmentation unit, 402... skeleton recognition unit, 403... skeleton coordinate calculation unit, 404... model generation unit, 405...hand coordinate calculation unit, 406...offset unit, 407...calculation unit, 408...coordinate conversion unit, 409...hand image generation unit, 410...presentation image generation unit, 411...first output unit, 412...calculation unit, 413...first difference calculation unit, 414...first integration unit, 415...calculation unit, 416...second difference calculation unit, 417...second integration unit, 418...initial position storage unit, 419...calculation unit, 420...second output unit

Claims

1. A remote control system for a robot having a manipulator, comprising: a first image capturing device that captures an image of an operator's hand; a position detection unit that detects the position of a specific part of the operator's hand from the image captured by the first image capturing device; a position determination unit that determines a target position for the manipulator based on the detected position of the specific part; and a control unit that controls the manipulator based on the determined target position.

2. The remote control system of claim 1, wherein the position detection unit comprises: a segmentation unit that extracts the specific portion from the image captured by the first imaging device; a skeleton recognition unit that recognizes a three-dimensional skeleton of the operator's hand from the image of the extracted specific portion; and a skeleton coordinate calculation unit that calculates the coordinates of the positions of the operator's fingertips and wrist in the three-dimensional skeleton in a coordinate system in the operator's space, and calculates the orientation of the operator's hand.

3. A remote control system as described in claim 1 or claim 2, comprising: a second camera capable of photographing the manipulator; and a presentation image generation unit that generates a presentation image to be presented to the operator by combining an image of the manipulator photographed by the second camera with an image of the manipulator photographed by the second camera by offsetting the coordinates of the operator's hand photographed by the first camera.

4. A remote control system as described in claim 3, comprising: a sensor that detects the joint angles of the manipulator; a hand coordinate calculation unit that calculates the coordinates of the fingertip positions of each finger of the manipulator and the coordinates of the wrist using an image of the manipulator captured by the second imaging device and the joint angles detected by the sensor; and a coordinate conversion unit that converts the coordinates of the operator's fingertip and wrist detected by the position detection unit by moving and rotating them based on offset values, wherein the presentation image generation unit generates the presentation image using the converted coordinates.

5. The remote control system described in claim 3, wherein the first image capturing device is a stereo camera or is composed of three or more cameras with at least a portion of their image capturing ranges overlapping, the second image capturing device is a stereo camera or is composed of three or more cameras, or is composed of three or more cameras with at least a portion of their image capturing ranges overlapping, and the distance between the operator's shoulder and the first image capturing device is equal to the distance between the base of the manipulator and the position of the second image capturing device.

6. The remote control system according to claim 4, wherein the coordinate conversion unit converts the coordinates of the operator's fingertips and wrist by moving and rotating them based on the coordinates after offsetting the wrist of the hand held by the manipulator, outputs the orientation of the operator's hand after the coordinate conversion, the position coordinates of the operator's fingertips after the coordinate conversion, and the offset value to be offset after the coordinate conversion to the presentation image generation unit, and outputs the position coordinates of the operator's fingertips after the coordinate conversion and the offset value after the coordinate conversion to the position determination unit.

7. A remote control device for remotely controlling a robot having a manipulator and controlling the manipulator, comprising: a position detection unit that detects the position of a specific part of the operator's hand from an image captured by a first image capture device that captures an image of the operator's hand; a position determination unit that determines a target position of the manipulator based on the detected position of the specific part; and an output unit that outputs the determined target position to the robot.

8. A remote control method for remotely operating a robot having a manipulator and controlling the manipulator, comprising: a position detection unit detects the position of a specific part of the operator's hand from an image captured by a first image capture device that captures an image of the operator's hand; a position determination unit determines a target position of the manipulator based on the detected position of the specific part; and an output unit outputs the determined target position to the robot.

9. A program that causes a computer of a remote control device that remotely controls a robot having a manipulator and controlling said manipulator to detect the position of a specific part of the operator's hand from an image captured by a first image capture device that captures an image of the operator's hand, determine a target position of the manipulator based on the detected position of the specific part, and output the determined target position to the robot.

Images