Remote operation system and remote operation method

Abstract

A remote operation system (100) according to the present disclosure comprises: an imaging device (2) capable of imaging a region including a target arrival point for a robot (1); a display device (82) for displaying a video imaged by the imaging device (2); an operation device (83) capable of receiving, from an operator, designation of a viewpoint and capable of receiving, from the operator, designation of a target position, which is an in-video position representing the target arrival point in the video for a remotely-operated machine; and a control device (6) for calculating the three-dimensional position of the arrival point by using a first target position corresponding to a target position which is in a first video imaged with a first viewpoint designated and is designated by the operator on the basis of the first video, a second target position corresponding to a target position which is in a second video imaged with a second viewpoint designated, is designated by the operator on the basis of the second video, and is designated by the operator on the basis of the second video of the target arrival point, and an imaging position, of the imaging device (2), corresponding to the second viewpoint.

Description

Remote control system and remote control method

[0001] The present disclosure relates to a remote control system and a remote control method for remotely controlling a remotely controlled machine.

[0002] In remote control systems that remotely operate robots and other machines, the operator operates the machine while understanding the situation around the machine through images captured by a camera mounted on the machine. In this case, it is desirable for the operator to be able to perceive not only two-dimensional information but also depth information.

[0003] Patent Document 1 discloses an indirect visual field presentation device that includes a motion detection unit that detects the motion of an operator and controls the position of a stereo camera mounted on a manipulator so as to be linked to the head motion of the operator detected by the motion detection unit. The indirect visual field presentation device described in Patent Document 1 displays, on a stereo display such as a head-mounted display, a stereoscopic image captured by a stereo camera that follows the head motion of the operator, and is therefore able to express motion parallax according to the motion of the operator.

[0004] International Publication No. 2016 / 152572

[0005] In the technology described in Patent Document 1, a three-dimensional image is displayed on a stereo display, allowing the operator to perceive depth. However, in the technology described in Patent Document 1, the operator must visually recognize the stereoscopic image on the stereo display, which increases the burden on the operator's brain.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a remote control system that allows the operator to perceive depth while reducing the burden on the operator's brain.

[0007] In order to solve the above-mentioned problems and achieve the object, a remote operation system according to the present disclosure includes an image capturing device capable of capturing an image of an area including a target destination of a remotely operated machine, a display device that displays an image captured by the image capturing device, and an operation device that can receive from an operator a designation of a viewpoint of the image capturing device and can receive from the operator a designation of a target position that is a position within the image that represents the target destination of the machine in the image. The remote operation system further includes a control device that calculates a three-dimensional position of the target destination using a first target position that is a target position within a first image that is designated by the operator based on a first image captured by the image capturing device when a first viewpoint is designated as the viewpoint, a second target position that is a target position within a second image that is designated by the operator based on a second image captured by the image capturing device when a second viewpoint is designated as the viewpoint, and the image capturing position of the image capturing device that corresponds to the second viewpoint.

[0008] According to the present disclosure, an effect is achieved in that the operator can perceive depth while reducing the burden on the operator's brain.

[0009] FIG. 1 is a diagram showing a configuration example of a remote control system according to the first embodiment. FIG. 2 is a diagram showing a configuration example of a robot according to the first embodiment. FIG. 3 is a diagram showing an example of an operation using a terminal device according to the first embodiment. FIG. 4 is a flowchart showing an example of a processing procedure in a control device according to the first embodiment. FIG. 5 is a diagram for explaining a method for calculating the three-dimensional position of a target arrival point according to the first embodiment. FIG. 6 is a diagram showing a configuration example of a remote control system according to the first embodiment when a plurality of monocular cameras are used. FIG. 7 is a diagram showing a configuration example of a remote control system according to the first embodiment when a 3D camera is used. FIG. 8 is a diagram showing a configuration example of a computer system that realizes the control device according to the first embodiment.

[0010] Hereinafter, a remote control system and a remote control method according to an embodiment will be described in detail with reference to the accompanying drawings.

[0011] First Embodiment. Fig. 1 is a diagram illustrating an example of the configuration of a remote operation system according to a first embodiment. The remote operation system 100 of this embodiment includes a terminal device 8 usable by an operator who remotely operates a robot 1, which is an example of a remotely operated machine; a control device 6 that identifies the three-dimensional position of a target arrival point of the robot 1; and an imaging device 2. The three-dimensional position is a position expressed in three dimensions and may be represented in a coordinate system fixed to the robot 1 (hereinafter also referred to as a robot-fixed coordinate system), a coordinate system fixed to the Earth (hereinafter also referred to as a fixed coordinate system), or any other coordinate system. At least one of the robot 1 and a robot control device 7 that controls the robot 1 may be included in the remote operation system 100.

[0012] The robot 1 includes a camera 2, a drive control unit 3, a drive mechanism 4, and a transmitter / receiver 5. The camera 2 can capture images of an area including a target arrival point of the robot 1, and can change the viewpoint by changing the capture position. As shown in FIG. 1 , the camera 2 includes a monocular camera 21, which is an example of a camera, a camera drive mechanism 23 that can change the capture position of the monocular camera 21, and a camera control unit 22 that controls the monocular camera 21 and the camera drive mechanism 23. The following mainly describes, as an example, a robot 1 that combines a cart and a robot arm (manipulator). However, the remotely controlled machine may be any remotely controlled device that includes a drive mechanism 4, and may be, but is not limited to, a mobile vehicle, a robot arm alone, a humanoid robot, or an industrial or household machine.

[0013] The transmitter / receiver 5 communicates with other devices. For example, the transmitter / receiver 5 can communicate with each of the terminal device 8, the control device 6, and the robot control device 7. For example, when the transmitter / receiver 5 receives a viewpoint change instruction from the control device 6 to change the viewpoint of a camera (the monocular camera 21 in this embodiment), the transmitter / receiver 5 notifies the camera control unit 22 of the received viewpoint change instruction. Note that while FIG. 1 illustrates an example in which the viewpoint change instruction is transmitted from the terminal device 8 to the robot 1 via the control device 6, this is not limiting and the instruction may be transmitted from the terminal device 8 to the robot 1. In this case, the terminal device 8 also transmits the viewpoint change instruction to the control device 6. The transmitter / receiver 5 also transmits video captured by the image capturing device 2 to the terminal device 8. The video captured by the image capturing device 2 may be received by the terminal device 8 via the control device 6. Furthermore, when the transmitter / receiver 5 receives a control command from the robot control device 7, the transmitter / receiver 5 outputs the received control command to the drive control unit 3.

[0014] The camera drive mechanism 23 in the image capture device 2 includes, for example, a device such as a camera arm that can rotate around a certain point within the robot 1 as a fulcrum, and the position of the monocular camera 21, i.e., the image capture position, is changed by rotating the camera arm with the monocular camera 21 attached to the device. The camera control unit 22 controls the camera drive mechanism 23 based on a viewpoint change instruction received from the transmitting / receiving unit 5. Furthermore, if the image capture device 2 is capable of changing the angle of view, the camera control unit 22 determines the angle of view in accordance with the angle of view setting instruction, and if the image capture device 2 is capable of changing the image capture direction (at least one of the pan and tilt angles), the camera control unit 22 changes the image capture direction in accordance with the image capture direction setting instruction.

[0015] FIG. 2 is a diagram showing an example of the configuration of a robot 1 according to this embodiment. The robot 1 shown in FIG. 2 includes a drive mechanism 4, which includes a dolly 41 and a robot arm 42. The robot arm 42 has a hand 43 at the tip thereof that can grasp an object or the like. The robot 1 can move by moving the dolly 41. In the example shown in FIG. 2, an L-shaped device that is part of the camera drive mechanism 23 is integrated with the dolly 41. A monocular camera 21 is attached to the upper end of the L-shaped device. As a result, the monocular camera 21 rotates around the fulcrum of the dolly 41 as the dolly 41 pivots (rotates). That is, in the example shown in FIG. 2, the drive mechanism that pivots the dolly 41 also serves as part of the camera drive mechanism 23.

[0016] In the example shown in FIG. 2 , in an xyz coordinate system in which the z-axis is vertical and the x-axis and y-axis are horizontal, the position of the monocular camera 21 moves as the monocular camera 21 rotates within the xy plane. This allows the robot 1 shown in FIG. 2 to change the viewpoint of the monocular camera 21. When grasping an object 9 by remote control, the robot 1 moves to the vicinity of the object 9, for example, as indicated by the arrow, and then grasps the object 9 with the hand 43. Note that FIG. 2 is merely an example, and the drive mechanism 4 and camera drive mechanism 23 of the robot 1 are not limited to the example shown in FIG. 2 . The camera drive mechanism 23 may be any mechanism capable of changing the viewpoint of the monocular camera 21. In the example shown in Figure 2, the shooting direction also changes in conjunction with the shooting position, so the viewpoint representing the viewing position and line of sight direction (shooting position and shooting direction) is determined according to the shooting position (position according to the rotation angle of the monocular camera 21). However, the method of changing the viewpoint of the monocular camera 21 is not limited to the example shown in Figure 2, and the shooting position and shooting direction may be set individually.

[0017] 1 and 2 show an example in which the image capturing device 2 is mounted on the robot 1, but this is not limiting and the image capturing device 2 may be provided outside the robot 1. For example, an image corresponding to a first viewpoint and an image corresponding to a second viewpoint may be generated by generating (combining) images from a pseudo-arbitrary viewpoint (e.g., the viewpoint of the robot 1) using images from multiple monocular cameras 21 fixed in the environment outside the robot 1. In this case, the viewpoint of the images may be changed by a command from the operator, and input of a target position for each image may be accepted. Alternatively, the robot 1 may be a robot arm 42 without a moving mechanism, and rails may be provided on walls or the like surrounding the robot arm 42, and a camera drive mechanism 23 that moves on the rails may be provided to change the viewpoint of the monocular camera 21.

[0018] In the example shown in FIG. 2 , the fixed axis of the robot arm 42 and the rotation axis of the dolly 41 are coaxial. The robot arm 42 can rotate around the fixed axis relative to the dolly 41. As a result, when the monocular camera 21 fixed to the dolly 41 (the position of the monocular camera 21 is offset from the dolly rotation axis by an L-shaped device) is moved in accordance with the rotation of the dolly 41, the position and orientation of the hand 43 of the robot arm 42 do not change. In this way, in the example shown in FIG. 2 , by rotating the dolly 41, the viewpoint of the monocular camera 21 can be changed while maintaining the position and orientation of the hand 43 of the robot arm 42, thereby simplifying control. Note that if the fixed axis of the robot arm 42 and the rotation axis of the dolly 41 are not coaxial, a control system can be devised to move the monocular camera 21 while maintaining the position and orientation of the hand 43. For example, the L-shaped device shown in FIG. 2 may rotate around the fixed axis of the robot arm 42 relative to the dolly 41.

[0019] Returning to the explanation of FIG. 1 , the terminal device 8 includes a transmitter / receiver 81, a display device 82, and an operation device 83. The transmitter / receiver 81 communicates with other devices. For example, the transmitter / receiver 81 receives camera footage captured by the monocular camera 21 of the imaging device 2, and outputs the received camera footage to the display device 82. The camera footage may be transmitted from the robot 1 to the terminal device 8 via the control device 6. The transmitter / receiver 81 transmits information received from the operation device 83 to the control device 6. The transmitter / receiver 81 may transmit part of the information received from the operation device 83 to the robot 1. The display device 82 displays the camera footage.

[0020] The operation device 83 accepts input operations from an operator remotely operating the robot 1. The operation device 83 can accept from the operator a designation of the viewpoint of the imaging device 2 and a designation of a target position, which is the position of a target arrival point in the camera image, from the operator. For example, the operation device 83 accepts an input of a viewpoint change instruction from the operator to change the viewpoint of the camera and outputs the viewpoint change instruction to the transceiver unit 81. The operation device 83 also accepts an input of a target position from the operator and outputs target position information indicating the target position to the transceiver unit 81. Note that the terminal device 8 converts the position on the screen of the terminal device 8 specified by the operation device 83 into a position in the real image, thereby representing the target position in the target position information as a position in the camera image. This allows the operation device 83 to designate the target position as a position in the camera image. The conversion of the position on the screen of the terminal device 8 into a position in the real image may be performed by the control device 6.

[0021] The terminal device 8 is, for example, but not limited to, a smartphone, a tablet, or a personal computer. The display device 82 and the operation device 83 of the terminal device 8 may be an integrated touch panel, or may be provided separately.

[0022] FIG. 3 is a diagram showing an example of an operation using the terminal device 8 of this embodiment. In the example shown in FIG. 3 , the terminal device 8 is a smartphone, a tablet, or the like, and a display device 82 and an operation device 83 are integrated together. As shown in FIG. 3 , while a camera image is displayed, the operator can specify a target point in the camera image as the target position 203. For example, the operator may specify the target position by tapping the target position 203. If the robot 1 has a hand 43, the target point is the target point of the hand 43. The target point is not limited to this, and may be, for example, the target point at the tip of the cart 41, or may be a target point at a specific location on the robot 1.

[0023] Furthermore, operation figures 201 and 202 are displayed on the display device 82. The operation figures 201 and 202 are figures indicating directions, with the operation figure 201 indicating a left direction and the operation figure 202 indicating a right direction. For example, when the operator taps the operation figure 201, the camera drive mechanism 23 rotates counterclockwise from the current position of the monocular camera 21 within the xy plane, thereby changing the viewpoint (shooting position and shooting direction) of the monocular camera 21. When the operator taps the operation figure 202, the camera drive mechanism 23 rotates clockwise from the current position of the monocular camera 21 within the xy plane, thereby changing the viewpoint of the monocular camera 21. In this example, tapping the operation figures 201 and 202 corresponds to a viewpoint change instruction. Note that the correspondence between the operation figures 201 and 202 and the rotation direction may be reversed. For example, the angle by which the camera drive mechanism 23 rotates with one tap on the operation figures 201 and 202 may be determined. Furthermore, the camera driving mechanism 23 may continuously rotate while the operation figures 201 and 202 are pressed. For example, the camera driving mechanism 23 may rotate while the operator is touching the operation figures 201 and 202, and the rotation of the camera driving mechanism 23 may stop when the operator stops touching. The shapes and arrangements of the operation figures 201 and 202 are not limited to the example shown in Fig. 3. Furthermore, the method of instructing a change in viewpoint is not limited to these examples, and for example, the operator may specify a rotation angle as a numerical value.

[0024] After specifying a target position once, the operator changes the viewpoint of the monocular camera 21 and specifies the target position while the camera image from the changed viewpoint is displayed. In this manner, in this embodiment, for one target point, the position of the target point in each of the camera images from different viewpoints is specified as the target position. For example, the operator specifies a first target position, which is the target position at a first viewpoint, and then specifies a second target position, which is the target position at a second viewpoint different from the first viewpoint. The first target position is a target position in the first image specified by the operator based on a first image captured by the image capture device 2 when the first viewpoint was specified, and the second target position is a target position in the second image specified by the operator based on a second image captured by the image capture device 2 when the second viewpoint was specified. For example, when the first viewpoint corresponds to the initial state of the monocular camera 21 and the shooting direction is determined in conjunction with the shooting position, the control device 6 of this embodiment calculates the three-dimensional position of the target arrival point using the first target position, the second target position, the initial position of the camera device 2 (monocular camera 21) (shooting position corresponding to the first viewpoint), and the shooting position of the camera device 2 corresponding to the second viewpoint. Although depth cannot be specified using only a camera image corresponding to one viewpoint, the control device 6 can thereby express depth by using the target positions specified for the two viewpoints. Note that when the shooting direction is determined in conjunction with the shooting position, the camera model includes information for determining the shooting direction corresponding to the shooting position, and the control device 6 determines the shooting direction using the camera model. For example, as shown in FIG. 2, when the camera drive mechanism 23 determines the shooting position based on the rotation angle, the camera drive mechanism 23 also determines the shooting direction based on the rotation angle.

[0025] Here, as illustrated in FIG. 2 , an example has been described in which the shooting direction is determined in conjunction with the shooting position. However, if the shooting direction is also specified, the shooting direction of the second viewpoint is also specified when the second viewpoint is specified. The control device 6 then calculates the three-dimensional position of the target point using the first target position, the second target position, the shooting position and shooting direction of the first viewpoint (predetermined initial shooting position and shooting direction), and the specified shooting position and shooting direction of the second viewpoint. Furthermore, if the first viewpoint is not set as the initial state and a shooting position is also specified for the first viewpoint, the shooting direction corresponding to the specified shooting position of the first viewpoint is used instead of the above-described initial shooting position and shooting direction. Furthermore, both the shooting position and shooting direction may be specified for the first viewpoint. The shooting position and shooting direction of the first viewpoint are information indicating the first viewpoint, and the shooting position and shooting direction of the second viewpoint are information indicating the second viewpoint. Therefore, the control device 6 only needs to calculate the three-dimensional position of the target point using the first target position, the second target position, and information indicating the first viewpoint and the second viewpoint. Furthermore, for example, the shooting position of the second viewpoint may be expressed with the shooting position of the first viewpoint as the origin. Details of the method for calculating the three-dimensional position of the target point will be described later.

[0026] Returning to the description of FIG. 1 , the control device 6 includes a transmitter / receiver 61, a model storage unit 62, and a 3D (three-dimensional) position identification unit 63. The transmitter / receiver 61 communicates with other devices. For example, when the transmitter / receiver 61 receives a viewpoint change instruction from the terminal device 8, it outputs the received viewpoint change instruction to the 3D position identification unit 63, and when the transmitter / receiver 61 receives a target position from the terminal device 8, it outputs the received target position to the 3D position identification unit 63. The transmitter / receiver 61 also outputs the three-dimensional position of the target arrival point identified by the 3D position identification unit 63 to the robot control device 7.

[0027] The model storage unit 62 stores a camera model used to calculate the three-dimensional position of the monocular camera 21. For example, the camera model includes a position model for calculating the position of the monocular camera 21, angle-of-view information indicating the angle of view of the monocular camera 21, and shooting direction information indicating the shooting direction. As described above, the position of the monocular camera 21 changes in response to a viewpoint change instruction, and therefore the position model is, for example, a calculation formula for calculating the position of the monocular camera 21 using parameters that change in response to the viewpoint change instruction.

[0028] For example, as shown in FIG. 2 , when the monocular camera 21 moves by rotating in a horizontal plane, the position model is a formula for calculating the position of the monocular camera 21 according to the rotation angle from the reference position. The reference position is, for example, a position corresponding to the initial state when no viewpoint change instruction has been received, but is not limited to this. For example, in the example shown in FIG. 2 , the monocular camera 21 rotates in accordance with the rotation of the camera arm without changing its vertical position. Therefore, the parameter that changes in accordance with the viewpoint change instruction is the rotation angle θ of the camera arm. If the direction from the fulcrum of the camera arm to the monocular camera 21 in the horizontal plane at the reference position is the −x direction and the length of the camera arm in the horizontal plane is r, the coordinate value of the position of the monocular camera 21 can be expressed as (−r × cos θ, −r × sin θ, h). The height h is the height of the monocular camera 21 in the vertical direction, i.e., the z direction. Note that the position of the monocular camera 21 may be expressed in either a robot-fixed coordinate system or a fixed coordinate system. The position model is not limited to this example, but may be determined depending on the drive method of the camera drive mechanism 23. The reference position is also not limited to the example described above.

[0029] If the angle of view of the monocular camera 21 is not changeable, the angle of view information is a fixed value. If the angle of view is changeable, the angle of view information may be changed in accordance with an angle of view setting instruction, or may be information for calculating the angle of view from a setting value corresponding to the setting instruction. If the shooting direction is fixed, i.e., if the pan and tilt angles of the monocular camera 21 are not changeable, the shooting direction information is a fixed value in a coordinate system fixed to the monocular camera 21. If the shooting direction is changeable, the shooting direction information may be changed in accordance with an shooting direction setting instruction, or may be information for calculating the shooting direction from a setting value corresponding to the setting instruction. The angle of view setting instruction and the shooting direction setting instruction are transmitted to the camera control unit 22 via the terminal device 8, the control device 6, and the transceiver 5, similar to the viewpoint change instruction. However, the method of transmitting the angle of view setting instruction and the shooting direction setting instruction to the camera control unit 22 is not limited thereto.

[0030] The 3D position identification unit 63 calculates the three-dimensional position of the target point using the viewpoint change instruction and target position received from the transmission / reception unit 61 and the camera model stored in the model storage unit 62, and outputs the calculation result to the transmission / reception unit 61. For example, the 3D position identification unit 63 calculates the three-dimensional position of the target point based on the principle of triangulation using the position of the monocular camera 21 at the first viewpoint before the monocular camera 21 is moved, the target position at the first viewpoint, the position of the monocular camera 21 at the second viewpoint after the monocular camera 21 is moved in response to the viewpoint change instruction, the target position at the second viewpoint, and the camera model. For example, as illustrated in FIG. 3 , if the viewpoint change instruction is to rotate the camera drive mechanism 23 by tapping the operation figures 201 and 202, the 3D position identification unit 63 can calculate the rotation angle from the reference position by storing the cumulative value of the change in angle caused by the tap. As a result, the 3D position identification unit 63 can calculate the position of the monocular camera 21 at the first viewpoint and the position of the monocular camera 21 at the second viewpoint using the rotation angle from the reference position and the position model. Details of the method for calculating the three-dimensional position of the target arrival point will be described later.

[0031] When the robot control device 7 receives the three-dimensional position of the target point from the control device 6, it generates a control command for operating the robot 1 based on the received three-dimensional position of the target point and transmits the generated control command to the robot 1. Note that the robot control device 7 may also receive operation information indicating instructions for operating the robot 1 in addition to the three-dimensional position of the target point from the terminal device 8, generate a control command based on the received operation information, and transmit the generated control command to the robot 1. For example, the robot control device 7 may receive operation information for operating the hand 43 of the robot 1 after the robot 1 reaches the target point, generate a control command for operating the hand 43 of the robot 1 based on the received operation information, and transmit the generated control command to the robot 1.

[0032] Although the robot control device 7 is provided separately from the control device 6 in FIG. 1 , this is not a limitation and the control device 6 may be provided within the robot control device 7. When the control device 6 is provided within the robot control device 7, the 3D position identification unit 63 may output the three-dimensional position of the target arrival point to a robot control unit (not shown) within the robot control device 7 without passing through the transceiver 61. Furthermore, while the control device 6 is provided separately from the terminal device 8 and the robot 1 in FIG. 1 , this is not a limitation and the control device 6 may be provided within the terminal device 8 or the robot 1. When the control device 6 is provided in the terminal device 8, the output from the operation device 83 may be input to the 3D position identification unit 63 without passing through the transceiver 81 and the transceiver 61. When the control device 6 is provided within the robot 1, the transceiver 5 may function as the transceiver 61. Furthermore, while the robot control device 7 is provided separately from the robot 1 in FIG. 1 , this is not a limitation and the robot control device 7 may be provided within the robot 1. When the robot control device 7 is provided inside the robot 1 , the control command output from the robot control device 7 may be input to the drive control unit 3 without going through the transmitter / receiver unit 5 .

[0033] Next, the operation of this embodiment will be described. FIG. 4 is a flowchart showing an example of a processing procedure in the control device 6 of this embodiment. As shown in FIG. 4, the control device 6 determines whether a target position has been specified (step S1). More specifically, the 3D position identification unit 63 determines whether the target position has been received from the transmission / reception unit 61. As described above, the operator operates the terminal device 8 to specify the position of the target point to be reached as the target position in the camera image. Upon receiving the target position input from the operator, the terminal device 8 transmits the target position to the control device 6. Note that FIG. 4 shows an example in which the initial viewpoint of the image capture device 2 is the first viewpoint, but the first viewpoint is not limited to the initial viewpoint of the image capture device 2, i.e., the initial shooting position and shooting direction. If the first viewpoint is not the initial viewpoint of the image capture device 2, the viewpoint is changed based on a specification from the operator before step S1. For example, if the shooting direction is linked to the shooting position, the viewpoint is determined once the shooting position is determined, and therefore the viewpoint is changed by specifying the shooting position. If the shooting position and shooting direction can be set separately, the viewpoint is specified by the operator by specifying the shooting position and shooting direction.

[0034] If the target position is not specified (step S1: No), the control device 6 repeats step S1. If the target position is specified (step S1: Yes), the control device 6 stores it as the target position in the first viewpoint (step S2). Specifically, the 3D position identification unit 63 stores the target position received from the transmission / reception unit 61 as the target position in the first viewpoint (hereinafter also referred to as the first target position).

[0035] Next, the control device 6 determines whether or not a viewpoint change instruction has been received (step S3). Specifically, the 3D position identification unit 63 determines whether or not a viewpoint change instruction has been received from the transmission / reception unit 61. As described above, the operator operates the terminal device 8 to input a viewpoint change instruction to the terminal device 8. Upon receiving the viewpoint change instruction input from the operator, the terminal device 8 transmits the viewpoint change instruction to the control device 6.

[0036] If a viewpoint change instruction has been received (step S3: Yes), the control device 6 instructs the camera 1 to change the viewpoint (step S4). Specifically, in step S4, the 3D position identification unit 63 transmits the viewpoint change instruction received from the transmission / reception unit 61 to the robot 1 via the transmission / reception unit 61.

[0037] Next, the control device 6 determines whether a target position has been specified (step S5). Specifically, the 3D position identification unit 63 determines whether a target position has been received from the transceiver 61, as in step S1. If a target position has not been specified (step S5: No), the control device 6 repeats the process from step S3. If a target position has been specified (step S5: Yes), the control device 6 stores the target position as a target position at the second viewpoint (step S6). Specifically, the 3D position identification unit 63 stores the target position received from the transceiver 61 as a target position at the second viewpoint (hereinafter also referred to as the second target position). As described above, the first target position and the second target position are the positions of the same target point, but the first target position indicates a position within the camera image captured at the first viewpoint, and the second target position indicates a position within the camera image captured at the second viewpoint.

[0038] Next, the control device 6 calculates the three-dimensional position of the target point (step S7). Specifically, the 3D position identification unit 63 calculates the three-dimensional position of the target point using the first target position, the second target position, and the camera model stored in the model storage unit 62, and outputs the calculated three-dimensional position to the transmission / reception unit 61.

[0039] FIG. 5 is a diagram illustrating a method for calculating the three-dimensional position of a target arrival point according to this embodiment. FIG. 5 is a diagram illustrating the monocular camera 21 viewed from above, showing an example in which the target arrival point of the robot 1 is the position of the target 300. Components of the robot 1 other than the monocular camera 21 are omitted from the illustration. The monocular camera 21 at the first viewpoint captures an image 301 by taking an image in a direction 400, and the monocular camera 21 at the second viewpoint captures an image 302 by taking an image in a direction 401. In the example shown in FIG. 5, the monocular camera 21 rotates in a rotation direction 303, thereby moving the monocular camera 21 from a position corresponding to the first viewpoint to a position corresponding to the second viewpoint. Note that the images 301 and 302 shown in FIG. 5 do not indicate that the images are actually positioned as shown, but merely illustrate the acquisition of images corresponding to a range equivalent to the angle of view of the monocular camera 21.

[0040] While checking the image 301 acquired from the first viewpoint, the operator specifies a position in the image 301 corresponding to the target 300 as a target position (first target position) using the terminal device 8. In the example shown in Fig. 5 , the target 300 is located to the right of the direction 400 at the first viewpoint, and therefore is also located to the right of the horizontal center in the image 301. Also, the target 300 is located to the right of the direction 401 at the second viewpoint, and therefore is also located to the right of the horizontal center in the image 302.

[0041] The 3D position identification unit 63 uses the camera model, the first target position specified in the image 301, and the second target position specified in the image 302 to calculate a target direction 402, which is the direction of the target 300 at a position corresponding to the first viewpoint, and a target direction 403, which is the direction of the target 300 at a position corresponding to the second viewpoint. The 3D position identification unit 63 uses the camera model to calculate the position corresponding to the first viewpoint and the position corresponding to the second viewpoint, and calculates the three-dimensional position of the target 300, i.e., the three-dimensional position of the target arrival point, by the principle of triangulation using the positions corresponding to the first viewpoint, the positions corresponding to the second viewpoint, the target direction 402, and the target direction 403. The rotation angle from the position corresponding to the first viewpoint to the position corresponding to the second viewpoint can be, for example, 90 degrees, but is not limited to this. Note that FIG. 5 is an example, and the movement direction and movement amount of the monocular camera 21 from the first viewpoint to the second viewpoint are not limited to the example shown in FIG. 5.

[0042] Returning to the description of Fig. 4, after step S7, the control device 6 outputs the calculated three-dimensional position (step S8), and ends the process. In step S8, more specifically, the transmitter / receiver 61 transmits the three-dimensional position received from the 3D position identification unit 63 to the robot control device 7.

[0043] If the answer is No in step S3, the control device 6 determines whether or not a target position has been designated, similarly to step S1 (step S9). If a target position has been designated (step S9: Yes), the control device 6 proceeds to step S6. If a target position has not been designated (step S9: No), the control device 6 repeats the process from step S3.

[0044] Through the above processing, the three-dimensional position of the target point is transmitted to the robot control device 7, and the robot control device 7 can operate the robot 1 based on the three-dimensional position of the target point. In this embodiment, the operator does not need to have a stereoscopic vision, but simply specifies a target position in each of the two camera images from different viewpoints while viewing the camera images displayed on the terminal device 8 as two-dimensional images. This reduces the burden on the operator's brain. Furthermore, while it is difficult to perceive depth, i.e., the shooting direction, from a single camera image, in this embodiment, the operator can view two camera images from different viewpoints, and therefore the operator can also perceive depth.

[0045] In the example described above, the viewpoint is changed by changing the position of the monocular camera 21. However, camera images corresponding to multiple viewpoints captured by the monocular camera 21 may be stored in a storage unit (not shown) of the control device 6 or in a device other than the control device 6. Then, when a predetermined condition is met, switching to processing using multiple camera images may be performed without driving the monocular camera 21. Furthermore, switching between displaying real images and stored images may be manually performed. Furthermore, the real images and stored images may be displayed in an overlapping manner. For example, the transparency of the stored images may be increased to display the real images and stored images in an overlapping manner. The overlapping display method is not limited to this example. The processing using multiple camera images is processing in which the control device 6 generates a three-dimensional image using the stored camera images. The predetermined condition may be, for example, a condition that a predetermined number of camera images have been stored, or a condition that the number of viewpoints corresponding to the stored camera images has reached a predetermined number, but is not limited thereto, and may be any condition that enables the generation of a three-dimensional image. The control device 6 transmits the generated 3D image to the terminal device 8, causing the display device 82 of the terminal device 8 to display the 3D image. This allows the 3D image to be presented to the operator, who can perform operations such as specifying a target position while viewing the 3D image. For example, the 3D image is displayed on a 3D monitor capable of stereoscopic display, and the operator specifies a target position in the 3D image using an operating means while sequentially changing the viewpoint. Alternatively, the control device 6 may perform processing using multiple camera images by using accumulated camera images and free viewpoint video technology to generate, as display images, an image corresponding to a first viewpoint specified by the operator and an image corresponding to a second viewpoint specified by the operator. The terminal device 8 displays these images and accepts the specification of target positions within each of these images, thereby performing processing similar to that described in FIG. 4.By performing processing using images from multiple cameras, it is possible to save power consumption compared to operating the camera drive mechanism 23 each time, and it is also possible to give people around the robot 1 a sense of security (it is possible to reduce the anxiety caused by the operation of the camera drive mechanism 23).

[0046] Although one monocular camera 21 is used in the example shown in Fig. 1, multiple monocular cameras 21 may be used. Fig. 6 is a diagram showing an example of the configuration of a remote control system of this embodiment when multiple monocular cameras 21 are used. The remote control system 100a shown in Fig. 6 is similar to the remote control system 100 shown in Fig. 1, except that it includes a robot 1a instead of the robot 1. The robot 1a is similar to the robot 1 shown in Fig. 1, except that it includes an image capturing device 2a instead of the image capturing device 2. Components having the same functions as those in the example shown in Fig. 1 are assigned the same reference numerals as in Fig. 1, and redundant explanations will be omitted.

[0047] In the example shown in FIG. 6 , the image capturing device 2 a includes multiple monocular cameras 21. In FIG. 6 , the image capturing device 2 a includes two monocular cameras 21, but the number of monocular cameras 21 may be three or more. The multiple monocular cameras 21 are installed at different positions on the robot 1 a. The positions of the monocular cameras 21 on the robot 1 a may be fixed or, as in the example shown in FIG. 1 , may be changeable by the camera driving mechanism 23. In the example shown in FIG. 6 , the terminal device 8 displays camera images captured by any of the multiple monocular cameras 21, and the operator specifies a target position in the displayed camera image. Switching between the monocular cameras 21 to be used may be performed automatically, for example, by automatically changing the monocular camera 21 to be used in response to a change in viewpoint through operation of the terminal device 8, or may be performed by the operator. For example, the operator may specify a first target position and a second target position by intentionally selecting and switching the monocular camera 21 to be used to specify the target position. Alternatively, the terminal device 8 may display multiple camera images taken by two monocular cameras 21, and the operator may input the target positions in the two camera images as the first target position and the second target position, respectively.

[0048] In addition, in the example shown in FIG. 1, a monocular camera 21 is used as the camera, but a 3D camera, which is an example of a camera and is called a depth camera, may be used instead of the monocular camera 21. FIG. 7 is a diagram showing an example of the configuration of a remote control system of this embodiment when a 3D camera is used. The remote control system 100b shown in FIG. 7 is similar to the remote control system 100 shown in FIG. 1 except that it includes a robot 1b instead of the robot 1. The robot 1b is similar to the robot 1 shown in FIG. 1 except that it includes an image capturing device 2b instead of the image capturing device 2. Components having the same functions as those in the example shown in FIG. 1 are assigned the same reference numerals as those in FIG. 1, and redundant explanations will be omitted.

[0049] In the example shown in FIG. 7 , the image capture device 2b includes a 3D camera 24 instead of the monocular camera 21. In the example shown in FIG. 7 , the terminal device 8 also displays depth information as a camera image. The terminal device 8 may display the depth information, for example, by changing color according to the depth (depth), or may display 3D data using the depth information so that the operator can perceive depth in response to the operator's movements to instruct a change in viewpoint. For example, by moving a part of the operator's body, such as the operator's head, or by the operator moving the terminal device 8 operated by the operator, the terminal device 8 may generate and display a 2D image similar to that obtained when the viewpoint is changed, based on the 3D data using the depth information. This image may be generated by the camera control unit 22 or may be controlled by the terminal device 8 (not shown). Although FIG. 7 illustrates a control device 6, if the 3D camera 24 functions as the control device 6, the remote operation system 100b may not include the control device 6. When the 3D camera 24 also functions as the control device 6, the 3D camera 24 functions as both a photographing device capable of acquiring three-dimensional information of the subject to be photographed and the control device 6.

[0050] When the 3D camera 24 is used, the 3D camera 24 acquires images from a fixed point, and the range in which depth can be detected is the visible range of the 3D camera 24. If the 3D camera 24 acquires images from a fixed point without moving, the camera drive mechanism 23 may not be provided. On the other hand, the operator may want to confirm depth beyond the visible range of the 3D camera 24, and the camera drive mechanism 23 may be provided to enable the position of the 3D camera 24 to be changed. For example, when the camera control unit 22 determines, based on a viewpoint change instruction received from the terminal device 8 via the transmission / reception unit 5, that the operator has specified an amount of change in viewpoint that exceeds the visible range of the 3D camera 24, the camera control unit 22 may control the camera drive mechanism 23 to change the position of the 3D camera 24 according to the amount of change in viewpoint. Note that if the 3D camera 24 starts moving after it has exceeded the visible range of the 3D camera 24, a delay occurs while the 3D camera 24 is moving. To avoid this delay, the camera control unit 22 may control the camera drive mechanism 23 to start moving the 3D camera 24 before it exceeds the visible range. For example, when the cumulative value of the amount of change in viewpoint reaches or exceeds a certain amount, the camera control unit 22 may control the camera drive mechanism 23 to start moving the 3D camera 24. Alternatively, the camera control unit 22 may control the camera drive mechanism 23 so that, when an instruction to change the viewpoint is given, the camera control unit 22 starts moving the 3D camera 24 in the instructed direction.

[0051] In addition, the image displayed by the terminal device 8 may be generated by the camera control unit 22 or the terminal device 8 so that a two-dimensional image is displayed on the terminal device 8 assuming that the viewpoint has moved according to the amount of change specified by the operator, regardless of whether the 3D camera 24 has moved or not.

[0052] Next, the hardware configuration of the control device 6 of this embodiment will be described. The control device 6 of this embodiment shown in FIGS. 1, 6, and 7 functions as the control device 6 by executing a computer program, which is a computer program that describes the processing to be performed by the control device 6, on the computer system. FIG. 8 is a diagram showing an example configuration of a computer system that realizes the control device 6 of this embodiment. As shown in FIG. 8, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

[0053] In FIG. 8 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processing of the control device 6 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 includes buttons, a keyboard, a mouse, a touchpad, and the like. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is, for example, a display such as an LCD (Liquid Crystal Display). Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that Fig. 8 is an example, and the configuration of the computer system is not limited to the example of Fig. 8. For example, the computer system that realizes the control device 6 does not need to include the input unit 102, the display unit 104, and the output unit 106.

[0054] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above configuration, for example, a computer program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as control device 6 of this embodiment in accordance with the program stored in storage unit 103.

[0055] In the above explanation, a program describing the processing in the control device 6 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0056] The 3D position identification unit 63 shown in Figures 1, 6, and 7 is realized by the control unit 101 shown in Figure 8 executing a computer program stored in the storage unit 103 shown in Figure 8. The storage unit 103 shown in Figure 8 is also used to realize the 3D position identification unit 63 shown in Figures 1, 6, and 7. The model storage unit 62 shown in Figures 1, 6, and 7 is part of the storage unit 103 shown in Figure 8. The transmission / reception unit 61 shown in Figures 1, 6, and 7 is realized by the communication unit 105 shown in Figure 8. Note that the control device 6 may be realized by multiple computer systems. Furthermore, for example, the control device 6 may be realized by a cloud computer system.

[0057] The terminal device 8 shown in Figures 1, 6 and 7 is also realized by, for example, the computer system shown in Figure 8. The display device 82 shown in Figures 1, 6 and 7 is realized by the display unit 104 shown in Figure 8. The operation device 83 shown in Figures 1, 6 and 7 is realized by the input unit 102 shown in Figure 8. The transmission / reception unit 81 shown in Figures 1, 6 and 7 is realized by the communication unit 105 shown in Figure 8.

[0058] 1, 6 and 7 may also be realized by, for example, the computer system shown in Fig. 8. At least a part of the camera control unit 22 and the drive control unit 3 shown in Fig. 1, 6 and 7 may be realized by a processing circuit including the control unit 101 and memory unit 103 shown in Fig. 8.

[0059] As described above, in this embodiment, camera images captured from the first and second viewpoints by cameras mounted on the robot 1 are displayed on the terminal device 8, and input of target positions, which are the positions of the target points to be reached in each camera image, is accepted. The accepted target positions are used to identify the three-dimensional position of the target point to be reached. This allows the operator to perceive depth while reducing the burden on the operator's brain. Furthermore, since the motion parallax of the monocular camera 21 is utilized, a display device 82 that displays two-dimensional images is sufficient, eliminating the need for dedicated hardware for stereoscopic vision, such as a head-mounted display or lenticular display. Furthermore, because the method involves specifying the target point of the robot 1, remote control is possible even when there is a communication delay. Furthermore, applying this embodiment can shorten work time, reduce operator fatigue, and improve work accuracy.

[0060] Second Embodiment Fig. 9 is a diagram showing an example of the configuration of a remote control system according to a second embodiment. A remote control system 100c according to this embodiment includes a terminal device 8a that can be used by an operator who remotely controls the robot 1, a control device 6a that identifies the three-dimensional position of the target arrival point of the robot 1, and an imaging device 2. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0061] The terminal device 8a is similar to the terminal device 8 of the first embodiment, except that a sensor 84 for detecting the operator's movements is added. The sensor 84 may be an acceleration sensor or a gyro sensor for detecting the acceleration, tilt, etc., of the terminal device 8a as the operator's movements, or a camera for capturing images of the operator to detect the movements of the operator's eyes, head, etc. The sensor 84 is not limited to the above-described example, as long as it can detect the operator's movements. In this embodiment, the operator changes the viewpoint of the monocular camera 21 through his / her own movements. That is, the viewpoint of the monocular camera 21 is switched depending on the operator's movements. For example, as in the first embodiment, the operator specifies a target position at the first viewpoint and then changes the viewpoint of the monocular camera 21 by shaking his / her head to the right. In the configuration example shown in FIG. 2 , for example, when the operator shakes his / her head to the right, the viewpoint of the monocular camera 21 is rotated by a predetermined angle by the camera drive mechanism 23. The correspondence between the operator's movements and how the viewpoint of the monocular camera 21 is changed may be predetermined, or may be changeable by the operator.

[0062] The movement for changing the viewpoint of the monocular camera 21 may be, for example, a movement using the terminal device 8, such as the operator tilting the terminal device 8, rotating the terminal device 8, or shaking the terminal device 8. Furthermore, the movement for changing the viewpoint of the monocular camera 21 may be, for example, a movement for changing the operator's line of sight, a movement for moving the operator's hand, a movement for changing the direction of the operator's face by shaking the operator's head, or a movement for moving the position of the operator's head. The movement for changing the viewpoint of the monocular camera 21 is not limited to these.

[0063] 9, the sensor 84 is provided inside the terminal device 8a, but if the sensor 84 is a camera that captures an image of the operator, the sensor 84 may be provided outside the terminal device 8a. The sensor information, which is the detection result of the sensor 84, is transmitted to the control device 6a by the transmitter / receiver 81.

[0064] The control device 6a is similar to the control device 6 of the first embodiment except for the addition of a conversion unit 64. When the transceiver 61 receives sensor information from the terminal device 8a, it outputs the received sensor information to the conversion unit 64. The conversion unit 64 converts the movement detected by the sensor 84 into a change amount of the viewpoint of the image capture device 2. For example, the conversion unit 64 converts the movement of the operator based on the sensor information into a viewpoint change instruction and outputs the viewpoint change instruction obtained by the conversion to the transceiver 61 and the 3D position identification unit 63. For example, the conversion unit 64 holds information indicating a correspondence between the movement of the operator and how to change the viewpoint of the monocular camera 21 as action correspondence information, and converts the movement of the operator detected from the sensor information into a viewpoint change instruction using the action correspondence information. The transceiver 61 transmits the viewpoint change instruction received from the conversion unit 64 to the robot 1.

[0065] The 3D position identification unit 63 calculates the three-dimensional position of the target arrival point, similarly to the first embodiment, using the target position received from the transmitting / receiving unit 61, the viewpoint change instruction received from the conversion unit 64, i.e., the amount of change in viewpoint converted by the conversion unit 64, and the camera model stored in the model storage unit 62. The operation of this embodiment other than that described above is the same as that of the first embodiment.

[0066] 9 shows an example in which the conversion unit 64 is provided in the control device 6a, but this is not limiting and the conversion unit 64 may be provided in the terminal device 8a. Also in this embodiment, the control device 6a may be provided in the terminal device 8a, in the robot control device 7, or in the robot 1. The control device 6a and the terminal device 8a of this embodiment are also realized by, for example, the computer system shown in FIG. 8, similar to the control device 6 and the terminal device 8.

[0067] When a plurality of monocular cameras 21 are used as shown in Fig. 6 of the first embodiment, the viewpoint may be switched in accordance with the movement of the operator using the terminal device 8a and the control device 6a of the present embodiment. When a plurality of 3D cameras 24 are used as shown in Fig. 7 of the first embodiment, the viewpoint may be switched in accordance with the movement of the operator using the terminal device 8a and the control device 6a of the present embodiment.

[0068] As described above, in this embodiment, the viewpoint of the monocular camera 21 is switched in conjunction with the movement of the operator, thereby improving the sense of realism of the operator's depth perception.

[0069] Third Embodiment. Fig. 10 is a diagram showing an example of the configuration of a remote control system according to a third embodiment. A remote control system 100d according to this embodiment includes a terminal device 8, a control device 6, and an image capturing device 2, similar to the first embodiment. In this embodiment, the image capturing device 2 is mounted on a robot 1c. The robot 1c is similar to the robot 1 according to the first embodiment, except that a collision sensor 10 for detecting a collision is added. The collision sensor 10 is, for example, a sensor that detects the possibility of a collision. Components having the same functions as those in the first embodiment are assigned the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0070] The robot 1c of this embodiment includes a robot arm 42 having a hand 43, similar to the robot 1 shown in FIG. 2 , and a collision sensor 10 is attached to the hand 43. The collision sensor 10 detects whether a collision is possible by determining whether the distance between the robot 1c and a surrounding object, wall, person, etc. is equal to or less than a certain value. For example, when the collision sensor 10 detects a collision, it may notify the drive control unit 3 of the collision possibility as a detection result, or it may notify the drive control unit 3 of the distance between the robot 1c and the surrounding object, wall, person, etc. as a detection result. The drive control unit 3 stops the operation of the robot 1c when it detects a collision of the robot 1c using the detection result of the collision sensor 10. For example, the drive control unit 3 may stop the operation of the robot 1c when it is notified by the collision sensor 10 of a collision possibility. Furthermore, when it is notified of the distance between the robot 1c and a surrounding object, wall, person, etc., it may stop the operation of the robot 1c when the distance is equal to or less than a certain value. This prevents the robot 1c from colliding with surrounding objects, walls, people, etc.

[0071] In the above example, the collision sensor 10 is added to the robot 1 in the configuration example shown in Fig. 1, but this is not limiting. The collision sensor 10 may be added to the robot 1a shown in Fig. 6 or the robot 1b shown in Fig. 7, and the drive control unit 3 may similarly stop the operation of the robots 1a and 1b when notified by the collision sensor 10 that there is a possibility of a collision. Furthermore, in the second embodiment, the collision sensor 10 may be added to the robot 1, and the drive control unit 3 may similarly stop the operation of the robot 1 when notified by the collision sensor 10 that there is a possibility of a collision.

[0072] Fourth Embodiment Fig. 11 is a diagram showing an example of a display screen according to a fourth embodiment. The configuration of the remote control system 100 according to this embodiment is the same as that of the remote control system 100 according to the first embodiment shown in Fig. 1. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0073] In this embodiment, the robot 1 includes a robot arm 42 having a hand 43, as illustrated in Fig. 2. In this embodiment, when the hand 43 reaches a target point, the camera image displayed on the terminal device 8 is automatically enlarged. The upper diagram in Fig. 11 shows the camera image displayed on the display device 82 of the terminal device 8 before enlargement. When the hand 43 reaches a target point, the camera image displayed on the display device 82 of the terminal device 8 is enlarged, as shown in the lower diagram in Fig. 11. This allows the operator to easily perform a grasping operation with the hand 43.

[0074] For example, when the robot 1 reaches the target point, the drive control unit 3 notifies the camera control unit 22 that the target point has been reached, and upon receiving the notification, the camera control unit 22 enlarges the camera image and transmits the enlarged camera image to the terminal device 8 via the transmission / reception unit 5. Alternatively, upon receiving the notification, the camera control unit 22 may instruct the terminal device 8 via the transmission / reception unit 5 to enlarge and display the camera image, and a display control unit (not shown) of the terminal device 8 may control the display to enlarge and display the camera image. The instruction to enlarge the camera image may be given via the control device 6.

[0075] In addition, in any one of the remote control system 100a shown in Figure 6, the remote control system 100b shown in Figure 7, the remote control system 100c described in embodiment 2, and the remote control system 100d described in embodiment 3, or a combination of two or more of these, the camera image displayed on the terminal device 8, 8a may be similarly enlarged.

[0076] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0077] Various aspects of the present disclosure are summarized below as appendices.

[0078] (Supplementary Note 1) A remote control system comprising: an image capture device capable of capturing an image of an area including a target point of a remote machine and capable of changing a viewpoint by changing the image capture position; a display device that displays an image captured by the image capture device; an operation device capable of receiving from an operator a designation of a viewpoint of the image capture device and a designation of a target position that is a position of the target point of the remote machine in the image from the operator; and a control device that calculates a three-dimensional position of the target point using a first target position that is the target position in the first image designated by the operator based on a first image captured by the image capture device with a first viewpoint designated as the viewpoint, a second target position that is the target position in the second image designated by the operator based on a second image captured by the image capture device with a second viewpoint designated as the viewpoint, and the image capture position of the image capture device corresponding to the second viewpoint. (Supplementary Note 2) The remote control system according to Supplementary Note 1, characterized in that the image capture device comprises: a monocular camera; and a camera drive mechanism that is capable of changing the image capture position of the monocular camera. (Supplementary Note 3) The remote control system according to Supplementary Note 1, wherein the image capturing device comprises a plurality of monocular cameras. (Supplementary Note 4) The remote control system according to Supplementary Note 1, wherein the image capturing device comprises a three-dimensional camera. (Supplementary Note 5) The remote control system according to Supplementary Note 4, wherein the image capturing device comprises a camera drive mechanism capable of changing the image capturing position of the three-dimensional camera. (Supplementary Note 6) The remote control system according to any one of Supplements 1 to 5, comprising: a sensor that detects the movement of an operator; and a conversion unit that converts the movement detected by the sensor into an amount of change in a viewpoint of the image capturing device, wherein the image capturing device changes the viewpoint based on the amount of change converted by the conversion unit, and the control device calculates the image capturing position of the image capturing device corresponding to the first viewpoint and the image capturing position of the image capturing device corresponding to the second viewpoint using the amount of change converted by the conversion unit.(Supplementary Note 7) The remote control system according to any one of Supplements 1 to 6, further comprising a collision sensor for detecting a collision, wherein the remote machine stops operation when it detects a collision of the remote machine using the detection result of the collision sensor. (Supplementary Note 8) The remote control system according to any one of Supplements 1 to 7, further comprising: the image displayed on the display device is enlarged when the remote machine reaches the target point. (Supplementary Note 9) The remote control system according to any one of Supplements 1 to 8, further comprising: the image capturing device is mounted on the remote machine. (Supplementary Note 10) The remote control system according to Supplementary Note 9, further comprising: a rotatable carriage and a robot arm rotatable around a fixed axis coaxial with the rotation axis of the carriage, wherein the camera of the image capturing device is fixed to the carriage and moves as the carriage rotates. (Supplementary Note 11) A remote control method for a remote control system equipped with a camera capable of capturing an image of an area including a target point of a remote machine and capable of changing a viewpoint by changing a camera position, comprising: a step of displaying an image captured by the camera; a step of accepting an input of a first target position, which is a position of the target point in the first image, specified by an operator based on a first image captured by the camera with a first viewpoint specified as the viewpoint; a step of accepting an input of a second target position, which is a position of the target point in the second image, specified by an operator based on a second image captured by the camera with a second viewpoint specified as the viewpoint; and a step of calculating a three-dimensional position of the target point using the first target position, the second target position, the camera position of the camera corresponding to the first viewpoint, and the camera position of the camera corresponding to the second viewpoint.

[0079] 1, 1a, 1b, 1c robot, 2, 2a, 2b imaging device, 3 drive control unit, 4 drive mechanism, 5, 61, 81 transmission / reception unit, 6, 6a control device, 7 robot control device, 8, 8a terminal device, 9 object, 10 collision sensor, 21 monocular camera, 22 camera control unit, 23 camera drive mechanism, 24 3D camera, 41 dolly, 42 robot arm, 43 hand, 62 model memory unit, 63 3D position identification unit, 64 conversion unit, 82 display device, 83 operation device, 84 sensor, 100, 100a, 100b, 100c, 100d remote operation system.

Claims

1. an imaging device capable of imaging an area including a target destination of a remotely controlled machine; a display device that displays the image captured by the imaging device; an operation device capable of receiving from an operator a designation of a viewpoint of the imaging device and a designation of a target position, which is a position within the image that represents the target arrival point of the machine in the image; a control device that calculates a three-dimensional position of the target arrival point using a first target position that is the target position in the first image that is designated by the operator based on a first image captured by the imaging device when a first viewpoint is designated as the viewpoint, a second target position that is the target position in the second image that is designated by the operator based on a second image captured by the imaging device when a second viewpoint is designated as the viewpoint, and an imaging position of the imaging device that corresponds to the second viewpoint; a sensor that detects the movement of an operator; a conversion unit that converts the movement detected by the sensor into a change amount of a viewpoint in the image capturing device; Equipped with The imaging device is A monocular camera and a camera drive mechanism capable of changing a viewpoint by changing the photographing position of the monocular camera; Equipped with the imaging device changes the viewpoint based on the change amount converted by the conversion unit, a control device that calculates a shooting position of the imaging device corresponding to the first viewpoint and a shooting position of the imaging device corresponding to the second viewpoint using the change amount converted by the conversion unit.

2. A photographing device capable of photographing an area including a target destination of a remotely operated machine; a display device that displays the image captured by the imaging device; an operation device capable of receiving from an operator a designation of a viewpoint of the imaging device and a designation of a target position, which is a position within the image that represents the target arrival point of the machine in the image; a control device that calculates a three-dimensional position of the target arrival point using a first target position that is the target position in the first image that is designated by the operator based on a first image captured by the imaging device when a first viewpoint is designated as the viewpoint, a second target position that is the target position in the second image that is designated by the operator based on a second image captured by the imaging device when a second viewpoint is designated as the viewpoint, and an imaging position of the imaging device that corresponds to the second viewpoint; a sensor that detects the movement of an operator; a conversion unit that converts the movement detected by the sensor into a change amount of a viewpoint in the image capturing device; Equipped with The imaging device is Multiple monocular cameras with different shooting positions, Equipped with the imaging device changes the viewpoint based on the change amount converted by the conversion unit, a control device that calculates a shooting position of the imaging device corresponding to the first viewpoint and a shooting position of the imaging device corresponding to the second viewpoint using the change amount converted by the conversion unit.

3. A collision sensor is provided to detect a collision, 2. The remote control system according to claim 1, wherein the machine stops operation when it detects a collision of the machine using the detection result of the collision sensor.

4. 2. The remote control system according to claim 1, wherein the image displayed on the display device is enlarged when the machine reaches the target point.

5. The remote control system according to claim 1 , wherein the image capturing device is mounted on the machine.

6. The machine includes a swivelable carriage and a robot arm rotatable about a fixed axis coaxial with the swivel axis of the carriage; 6. The remote control system according to claim 5, wherein the camera of the photographing device is fixed to the dolly and moves as the dolly turns.

7. an imaging device capable of imaging an area including a target destination of a remotely controlled machine; a display device that displays the image captured by the imaging device; an operation device capable of receiving from an operator a designation of a viewpoint of the imaging device and a designation of a target position, which is a position within the image that represents the target arrival point of the machine in the image; a control device that calculates a three-dimensional position of the target arrival point using a first target position that is the target position in the first image that is designated by the operator based on a first image captured by the imaging device when a first viewpoint is designated as the viewpoint, a second target position that is the target position in the second image that is designated by the operator based on a second image captured by the imaging device when a second viewpoint is designated as the viewpoint, and an imaging position of the imaging device that corresponds to the second viewpoint; Equipped with The imaging device is A monocular camera and a camera drive mechanism capable of changing a viewpoint by changing the photographing position of the monocular camera; Equipped with When a predetermined condition is satisfied, the control device generates a display image to be displayed on the display device using the stored plurality of images; the display device displays the display image; A remote control system characterized in that the operation device accepts designation of a position within the displayed image that represents the target arrival point of the machine.

8. A remote control method for a remote control system including a camera device capable of photographing an area including a target arrival point of a remotely operated machine, the camera device having a plurality of monocular cameras with different photographing positions and capable of changing the viewpoint by changing the photographing position, and a sensor that detects the movement of an operator, displaying the image captured by the imaging device; receiving an input of a first target position, which is a position of the target arrival point in the first image, designated by an operator based on a first image captured by the image capturing device in a state in which a first viewpoint is designated as the viewpoint; receiving an input of a second target position, which is a position of the target arrival point in the second image, designated by an operator based on a second image captured by the image capturing device in a state in which a second viewpoint is designated as the viewpoint; converting the movement detected by the sensor into a change in viewpoint of the image capture device; calculating a shooting position of the image capturing device corresponding to the first viewpoint and a shooting position of the image capturing device corresponding to the second viewpoint using the converted change amount; calculating a three-dimensional position of the target arrival point using the first target position, the second target position, a photographing position of the photographing device corresponding to the first viewpoint, and a photographing position of the photographing device corresponding to the second viewpoint; A remote control method comprising: