Remote control system and remote control method

Abstract

A remote operation system (100) according to the present disclosure comprises: an imaging device (2) capable of imaging a region including a target arrival point for a robot (1); a display device (82) for displaying a video imaged by the imaging device (2); an operation device (83) capable of receiving, from an operator, designation of a viewpoint and capable of receiving, from the operator, designation of a target position, which is an in-video position representing the target arrival point in the video for a remotely-operated machine; and a control device (6) for calculating the three-dimensional position of the arrival point by using a first target position corresponding to a target position which is in a first video imaged with a first viewpoint designated and is designated by the operator on the basis of the first video, a second target position corresponding to a target position which is in a second video imaged with a second viewpoint designated, is designated by the operator on the basis of the second video, and is designated by the operator on the basis of the second video of the target arrival point, and an imaging position, of the imaging device (2), corresponding to the second viewpoint.

Description

Technical Field

[0006] , ,

[0005] , , ,

[0001] The present disclosure relates to a remote operation system and a remote operation method for remotely operating a remotely operated machine.

Background Art

[0002] In a remote operation system for remotely operating a machine such as a robot, an operator operates while grasping the situation around the machine based on the image of a camera mounted on the remotely operated machine. At this time, it is desirable that the operator can perceive not only planar information but also depth.

[0003] Patent Document 1 discloses an indirect vision presentation device that includes a motion detection unit for detecting the motion of an operator and controls the position of a stereo camera mounted on a manipulator so as to be linked to the motion of the head of the operator detected by the motion detection unit. The indirect vision presentation device described in Patent Document 1 displays a stereoscopic image captured by a stereo camera that follows the motion of the head of the operator on a stereo display such as a head-mounted display, and thus can express a motion parallax corresponding to the motion of the operator.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique described in Patent Document 1, since a stereoscopic image is displayed on a stereo display, the operator can perceive depth. However, in the technique described in Patent Document 1, the operator needs to perform stereoscopic recognition by visually recognizing the stereo display. Therefore, the mental load on the operator increases.

[0006] This disclosure is made in view of the above, and aims to provide a remote control system that reduces the operator's mental load while also enabling the operator to perceive depth. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the remote control system according to this disclosure comprises: a camera capable of photographing an area including the target destination of a remotely controlled machine; a display device that displays the image captured by the camera; and an operating device that can receive a designation of the viewpoint of the camera from the operator and can receive a designation of a target position, which is the position in the image representing the target destination of the machine in the image. The remote control system further comprises: a control device that calculates the three-dimensional position of the target destination using a first target position, which is the target position in the first image specified by the operator based on a first image captured by the camera with a first viewpoint specified as the viewpoint; a second target position, which is the target position in the second image specified by the operator based on a second image captured by the camera with a second viewpoint specified as the viewpoint; and the shooting position of the camera corresponding to the second viewpoint; a sensor that detects the operator's movement; and a conversion unit that converts the movement detected by the sensor into an amount of change in the viewpoint of the camera. The imaging device comprises a monocular camera and a camera drive mechanism capable of changing the viewpoint by changing the shooting position of the monocular camera, and the viewpoint is changed based on the amount of change converted by the conversion unit. The shooting position of the shooting device corresponding to the first viewpoint is predetermined or calculated based on the first viewpoint received by the operating device. The control device is The shooting position of the shooting device corresponding to the first viewpoint and the movement detected by the sensor after the first target position has been specified. Amount of change converted by the conversion unit and, Using , the The shooting position of the shooting device corresponding to the two viewpoints Place Calculate. [Effects of the Invention]

[0008] According to this disclosure, the system reduces the operator's brain load while also allowing the operator to perceive depth. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of the remote control system according to Embodiment 1. [Figure 2] A diagram showing an example configuration of the robot in Embodiment 1. [Figure 3] This figure shows an example of operation using the terminal device of Embodiment 1. [Figure 4] A flowchart showing an example of the processing procedure in the control device of Embodiment 1. [Figure 5] A diagram illustrating the method for calculating the three-dimensional position of the target point in Embodiment 1. [Figure 6] This figure shows an example configuration of the remote control system of Embodiment 1 when using multiple monocular cameras. [Figure 7] This figure shows an example configuration of the remote control system of Embodiment 1 when using a 3D camera. [Figure 8] This figure shows an example configuration of a computer system that implements the control device of Embodiment 1. [Figure 9] This figure shows an example configuration of the remote control system according to Embodiment 2. [Figure 10] This figure shows an example configuration of the remote control system according to Embodiment 3. [Figure 11] A diagram showing an example of the display screen according to Embodiment 4. [Modes for carrying out the invention]

[0010] The remote control system and remote control method according to the embodiment will be described in detail below with reference to the drawings.

[0011] Embodiment 1. Figure 1 shows an example of the configuration of a remote control system according to Embodiment 1. The remote control system 100 of this embodiment includes a terminal device 8 that can be used by an operator who remotely controls a robot 1, which is an example of a machine to be remotely controlled; a control device 6 that identifies the three-dimensional position of the target destination of the robot 1; and a camera device 2. The three-dimensional position is a position expressed in three dimensions and may be expressed in a coordinate system fixed to the robot 1 (hereinafter also referred to as the robot fixed coordinate system), a coordinate system fixed to the Earth (hereinafter also referred to as the fixed coordinate system), or a coordinate system other than these. At least one of the robot 1 and the robot control device 7 that controls the robot 1 may be included in the remote control system 100.

[0012] Robot 1 comprises a camera 2, a drive control unit 3, a drive mechanism 4, and a transmitting / receiving unit 5. The camera 2 is capable of photographing an area including the target point reached by Robot 1, and the viewpoint can be changed by changing the shooting position. As shown in Figure 1, the camera 2 comprises a monocular camera 21, which is an example of a camera, a camera drive mechanism 23 that can change the shooting position of the monocular camera 21, and a camera control unit 22 that controls the monocular camera 21 and the camera drive mechanism 23. In the following, as an example, we will mainly describe an example in which the remotely controlled machine is Robot 1, which is a combination of a cart and a robot arm (manipulator), but the remotely controlled machine may be any remotely controllable device equipped with a drive mechanism 4, and may be a mobile vehicle, a robot arm alone, a humanoid robot, an industrial or household machine, etc., and is not limited to these.

[0013] The transmission / reception unit 5 communicates with other devices. For example, the transmission / reception unit 5 can communicate with the terminal device 8, the control device 6, and the robot control device 7 respectively. When the transmission / reception unit 5 receives a viewpoint change instruction for instructing to change the viewpoint of a camera (a monocular camera 21 in this embodiment) from the control device 6, it notifies the received viewpoint change instruction to the camera control unit 22. In FIG. 1, an example is shown where the viewpoint change instruction is transmitted from the terminal device 8 to the robot 1 via the control device 6, but it is not limited to this, and it may be transmitted from the terminal device 8 to the robot 1. In this case, the terminal device ⑧ also transmits a viewpoint change instruction to the control device 6. Further, the transmission / reception unit 5 transmits the video acquired by the imaging device 2 to the terminal device 8. The video acquired by the imaging device 2 may be received by the terminal device 8 via the control device 6. Also, when the transmission / reception unit 5 receives a control command from the robot control device 7, it outputs the received control command to the drive control unit 3.

[0014] The camera drive mechanism 23 in the imaging device 2 includes, for example, an instrument such as a camera arm that can rotate about a point within the robot 1, and changes the position of the monocular camera 21, that is, the imaging position, by rotating the camera arm with the monocular camera 21 attached to the instrument. The camera control unit 22 controls the camera drive mechanism 23 based on the viewpoint change instruction received from the transmission / reception unit 5. Also, when the imaging device 2 can change the angle of view, the camera control unit 22 determines the angle of view according to the setting instruction of the angle of view, and when the imaging device 2 can change the imaging direction (at least one of the pan and tilt angles), the camera control unit 22 changes the imaging direction according to the setting instruction of the imaging direction.

[0015] FIG. 2 is a diagram showing a configuration example of the robot 1 of the present embodiment. The robot 1 shown in FIG. 2 includes, as a drive mechanism 4, a carriage 41 and a robot arm 42, and a hand (gripper) 43 capable of gripping an object or the like is provided at the tip of the robot arm 42. The robot 1 can move by the movement of the carriage 41. Also, in the example shown in FIG. 2, an L-shaped device that is part of the camera drive mechanism 23 is integrated with the carriage 41. A monocular camera 21 is attached to the upper end of the L-shaped device. As a result, the monocular camera 21 rotates around the fulcrum of the carriage 41 as the carriage 41 turns (rotates). That is, in the example shown in FIG. 2, the drive mechanism for turning the carriage 41 also serves as part of the camera drive mechanism 23.

[0016] In the example shown in FIG. 2, in the xyz coordinate system where the z-axis is in the vertical direction and the x-axis and y-axis are in the horizontal plane, the position of the monocular camera 21 moves by the rotation of the monocular camera 21 in the xy plane. As a result, the robot 1 shown in FIG. 2 can change the viewpoint of the monocular camera 21. When gripping the object 9 by remote operation, the robot 1, for example, moves to the vicinity of the object 9 as shown by the arrow and then grips the object 9 with the hand 43. Note that FIG. 2 is an example, and the drive mechanism 4 and the camera drive mechanism 23 of the robot 1 are not limited to the example shown in FIG. 2. The camera drive mechanism 23 may be any mechanism capable of changing the viewpoint of the monocular camera 21. In the example shown in FIG. 2, since the shooting direction also changes in conjunction with the shooting position, the viewpoint representing the viewing position and the line-of-sight direction (shooting position and shooting direction) is determined according to the shooting position (the position corresponding to the rotation angle of the monocular camera 21). However, the method of changing the viewpoint of the monocular camera 21 is not limited to the example shown in FIG. 2, and the shooting position and the shooting direction may be set individually.

[0017] In Figures 1 and 2, an example is shown in which the imaging device 2 is mounted on the robot 1. However, the imaging device 2 is not limited to this configuration and may be located outside the robot 1. For example, images corresponding to the first viewpoint and images corresponding to the first viewpoint may be generated by using images from multiple monocular cameras 21 fixed in the environment outside the robot 1 to generate (combine) images of a pseudo-arbitrary viewpoint (e.g., the viewpoint of the robot 1). In this case as well, the viewpoint of the images can be changed by the operator's specification, and input of the target position in each image can be accepted. Alternatively, if the robot 1 is a robot arm 42 without a movement mechanism, the viewpoint of the monocular camera 21 can be changed by providing rails on the wall or other surface around the robot arm 42 and providing a camera drive mechanism 23 that moves along the rails.

[0018] Furthermore, in the example shown in Figure 2, the fixed axis of the robot arm 42 and the pivot axis of the trolley 41 are coaxial. The robot arm 42 can rotate around the fixed axis relative to the trolley 41. As a result, when the monocular camera 21 (the position of the monocular camera 21 is offset from the trolley's pivot axis by an L-shaped device) fixed to the trolley 41 is moved in conjunction with the rotation of the trolley 41, the position and orientation of the end-effector 43 of the robot arm 42 do not change. Thus, in the example shown in Figure 2, by rotating the trolley 41, the viewpoint of the monocular camera 21 can be changed while maintaining the position and orientation of the end-effector 43 of the robot arm 42, thereby simplifying control. If the fixed axis of the robot arm 42 and the pivot axis of the trolley 41 are not coaxial, the control system must be designed to move the monocular camera 21 while maintaining the position and orientation of the end-effector 43. For example, the L-shaped device shown in Figure 2 may rotate around the fixed axis of the robot arm 42 relative to the trolley 41.

[0019] Returning to the explanation of Figure 1, the terminal device 8 comprises a transmitting / receiving unit 81, a display device 82, and an operating device 83. The transmitting / receiving unit 81 communicates with other devices. For example, the transmitting / receiving unit 81 receives camera images, which are images captured by the monocular camera 21 of the imaging device 2, and outputs the received camera images to the display device 82. The camera images may also be transmitted from the robot 1 to the terminal device 8 via the control device 6. The transmitting / receiving unit 81 transmits information received from the operating device 83 to the control device 6. The transmitting / receiving unit 81 may also transmit some of the information received from the operating device 83 to the robot 1. The display device 82 displays the camera images.

[0020] The control device 83 receives input from an operator remotely controlling the robot 1. The control device 83 can receive the operator's specification of the viewpoint of the camera device 2, and can also receive the operator's specification of the target position, which is the position of the target destination in the camera image. For example, the control device 83 receives input of a viewpoint change instruction from the operator to change the camera's viewpoint, and outputs the viewpoint change instruction to the transmitting / receiving unit 81. The control device 83 also receives input of the target position from the operator and outputs target position information indicating the target position to the transmitting / receiving unit 81. The terminal device 8 converts the position on the terminal device 8's screen specified by the control device 83 to the position on the actual image, thereby representing the target position in the target position information as a position in the camera image. This allows the control device 83 to specify the target position as a position in the camera image. The conversion of the position on the terminal device 8's screen to the position on the actual image may be performed by the control device 6.

[0021] The terminal device 8 is, for example, a smartphone, a tablet, or a personal computer, but is not limited to these. The display device 82 and the operating device 83 in the terminal device 8 may be an integrated touch panel, or they may be provided separately.

[0022] Figure 3 shows an example of operation using the terminal device 8 of this embodiment. In the example shown in Figure 3, the terminal device 8 is a smartphone, tablet, etc., and the display device 82 and the operating device 83 are integrated. As shown in Figure 3, the operator can specify the target destination in the camera image as the target position 203 while the camera image is displayed. For example, the operator may specify the target position by tapping the target position 203. If the robot 1 has an end effector 43, the target destination is the target destination of the end effector 43. The target destination is not limited to this, and may be the target destination of the tip of the trolley 41, or any target destination of a specific part of the robot 1.

[0023] Furthermore, the display device 82 also displays operation shapes 201 and 202. Operation shapes 201 and 202 are shapes that indicate direction, with operation shape 201 indicating the left direction and operation shape 202 indicating the right direction. For example, when the operator taps operation shape 201, the camera drive mechanism 23 rotates counterclockwise in the xy plane from the current position of the monocular camera 21, thereby changing the viewpoint (shooting position and shooting direction) of the monocular camera 21. When the operator taps operation shape 202, the camera drive mechanism 23 rotates clockwise in the xy plane from the current position of the monocular camera 21, thereby changing the viewpoint of the monocular camera 21. In this example, tapping operation shapes 201 and 202 corresponds to an instruction to change the viewpoint. Note that the correspondence between operation shapes 201 and 202 and the direction of rotation may be reversed. For example, the angle by which the camera drive mechanism 23 rotates with a single tap of operation shapes 201 and 202 may be defined. Furthermore, the camera drive mechanism 23 may rotate continuously while the operation shapes 201 and 202 are pressed. For example, the camera drive mechanism 23 may rotate while the operator is touching operation shapes 201 and 202, and the rotation of the camera drive mechanism 23 may stop when the operator stops touching them. The shape and arrangement of operation shapes 201 and 202 are not limited to the examples shown in Figure 3. Also, the method of instructing a change in viewpoint is not limited to these examples; for example, the rotation angle may be specified numerically by the operator.

[0024] The operator first specifies the target position, then changes the viewpoint of the monocular camera 21, and then specifies the target position while the camera image from the changed viewpoint is displayed. In this way, in this embodiment, for a single target destination, the position of the target destination in each camera image from different viewpoints is specified as the target position. For example, the operator specifies the first target position, which is the target position in the first viewpoint, and then specifies the second target position, which is the target position in the second viewpoint, which is different from the first viewpoint. The first target position is the target position in the first image specified by the operator based on the first image captured by the shooting device 2 with the first viewpoint specified, and the second target position is the target position in the second image specified by the operator based on the second image captured by the shooting device 2 with the second viewpoint specified. For example, if the first viewpoint corresponds to the initial state of the monocular camera 21 and the shooting direction is determined in conjunction with the shooting position, the control device 6 of this embodiment calculates the three-dimensional position of the target destination using the first target position, the second target position, the initial position of the shooting device 2 (monocular camera 21) (shooting position corresponding to the first viewpoint), and the shooting position of the shooting device 2 corresponding to the second viewpoint. While depth cannot be specified using only the camera image corresponding to one viewpoint, the control device 6 can thus represent depth by using the target positions specified for each of the two viewpoints. Furthermore, if the shooting direction is determined in conjunction with the shooting position, the camera model is assumed to contain information for determining the shooting direction corresponding to the shooting position, and the control device 6 determines the shooting direction using the camera model. For example, as shown in Figure 2, if the shooting position is determined by the rotation angle of the camera drive mechanism 23, the shooting direction is also determined according to the rotation angle of the camera drive mechanism 23.

[0025] Here, we have explained an example where the shooting direction is determined in conjunction with the shooting position, as illustrated in Figure 2. However, if the shooting direction is also specified, the shooting direction of the second viewpoint is also specified when the second viewpoint is specified. The control device 6 then calculates the 3D position of the target destination using the first target position, the second target position, the shooting position and shooting direction of the first viewpoint (a predetermined initial shooting position and shooting direction), and the specified shooting position and shooting direction of the second viewpoint. Furthermore, if the shooting position of the first viewpoint is also specified, rather than being the initial state, the specified shooting position and shooting direction of the first viewpoint are used instead of the initial shooting position and shooting direction described above. In addition, both the shooting position and shooting direction may be specified for the first viewpoint as well. Shooting position and shooting direction of the first viewpoint teeth The first viewpoint is represented by information indicating the first viewpoint, and the shooting position and direction of the second viewpoint are also represented by information indicating the second viewpoint. Therefore, the control device 6 only needs to calculate the three-dimensional position of the target destination using the first target position, the second target position, and the information indicating the first and second viewpoints. For example, the shooting position of the second viewpoint may be represented with the shooting position of the first viewpoint as the origin. Details of the method for calculating the three-dimensional position of the target destination will be described later.

[0026] Returning to the explanation of Figure 1, the control device 6 comprises a transmitting / receiving unit 61, a model storage unit 62, and a 3D (three-dimensional) position identification unit 63. The transmitting / receiving unit 61 communicates with other devices. For example, when the transmitting / receiving unit 61 receives a viewpoint change instruction from the terminal device 8, it outputs the received viewpoint change instruction to the 3D position identification unit 63. When it receives a target position from the terminal device 8, it outputs the received target position to the 3D position identification unit 63. The transmitting / receiving unit 61 also outputs the three-dimensional position of the target destination, identified by the 3D position identification unit 63, to the robot control device 7.

[0027] The model storage unit 62 stores a camera model used to calculate the three-dimensional position of the monocular camera 21. For example, the camera model includes a position model for calculating the position of the monocular camera 21, angle of view information indicating the angle of view of the monocular camera 21, and shooting direction information indicating the shooting direction. As described above, the position of the monocular camera 21 is changed in response to a viewpoint change instruction, so the position model is a calculation formula for calculating the position of the monocular camera 21 using parameters that are changed in response to the viewpoint change instruction.

[0028] For example, as illustrated in Figure 2, when the monocular camera 21 moves by rotating in the horizontal plane, the position model is a formula for calculating the position of the monocular camera 21 according to its rotation angle from the reference position. The reference position is, for example, the position corresponding to the initial state when no viewpoint change instruction has been received, but is not limited to this. For example, in the example shown in Figure 2, the monocular camera 21 rotates along with the rotation of the camera arm without changing its vertical position. Therefore, the parameter that changes in response to the viewpoint change instruction is the rotation angle θ of the camera arm. If the direction from the pivot point of the camera arm toward the monocular camera 21 in the horizontal plane at the reference position is the -x direction, and the length of the camera arm in the horizontal plane is r, then the coordinate value of the position of the monocular camera 21 can be expressed as (-r × cosθ, -r × sinθ, h). The height h is the height of the monocular camera 21 in the vertical direction, i.e., the z direction. The position of the monocular camera 21 may be expressed in the robot fixed coordinate system or in the fixed coordinate system. The position model is not limited to this example and may be determined according to the driving method of the camera driving mechanism 23. Similarly, the reference position is not limited to the example described above.

[0029] If the field of view of the monocular camera 21 cannot be changed, the field of view information is a fixed value. If the field of view can be changed, the field of view information may be changed according to the field of view setting instruction, or it may be information for calculating the field of view from the setting value corresponding to the setting instruction. If the shooting direction is fixed, that is, if the pan and tilt angles of the monocular camera 21 cannot be changed, the shooting direction information is a fixed value in the coordinate system fixed to the monocular camera 21. If the shooting direction can be changed, the shooting direction information may be changed according to the shooting direction setting instruction, or it may be information for calculating the shooting direction from the setting value corresponding to the setting instruction. The field of view setting instruction and the shooting direction setting instruction are transmitted to the camera control unit 22 via the terminal device 8, the control device 6, and the transmitting / receiving unit 5, similar to the viewpoint change instruction, but the method of transmitting the field of view setting instruction and the shooting direction setting instruction to the camera control unit 22 is not limited to this.

[0030] The 3D position identification unit 63 uses the viewpoint change instruction and target position received from the transmitting / receiving unit 61 and the camera model stored in the model storage unit 62 to calculate the 3D position of the target destination and outputs the calculation result to the transmitting / receiving unit 61. For example, the 3D position identification unit 63 uses the position of the monocular camera 21 in the first viewpoint before the monocular camera 21 moves, the target position in the first viewpoint, the position of the monocular camera 21 in the second viewpoint after the monocular camera 21 moves due to the viewpoint change instruction, the target position in the second viewpoint, and the camera model to calculate the 3D position of the target destination using the principle of triangulation. For example, as illustrated in Figure 3, if the viewpoint change instruction causes the camera drive mechanism 23 to rotate by tapping the operation shapes 201 and 202, the 3D position identification unit 63 can calculate the rotation angle from the reference position by storing the cumulative value of the angle change amount changed by the tap. As a result, the 3D position identification unit 63 can use the rotation angle from the reference position and the position model to calculate the position of the monocular camera 21 at the first viewpoint and the position of the monocular camera 21 at the second viewpoint. Details of the method for calculating the 3D position of the target destination will be described later.

[0031] When the robot control device 7 receives the 3D position of the target destination from the control device 6, it generates a control command to operate the robot 1 based on the received 3D position of the target destination and transmits the generated control command to the robot 1. In addition, the robot control device 7 may also receive operation information from the terminal device 8 indicating instructions to operate the robot 1, generate a control command based on the received operation information, and transmit the generated control command to the robot 1. For example, after the robot 1 reaches the target destination, the robot control device 7 may receive operation information to operate the end effector 43 of the robot 1, generate a control command to operate the end effector 43 of the robot 1 based on the received operation information, and transmit the generated control command to the robot 1.

[0032] In Figure 1, the robot control device 7 is provided separately from the control device 6, but the system is not limited to this configuration, and the control device 6 may be provided within the robot control device 7. If the control device 6 is provided within the robot control device 7, the 3D position identification unit 63 may output the 3D position of the target destination to a robot control unit (not shown) within the robot control device 7 without going through the transmitting / receiving unit 61. Also, in Figure 1, the control device 6 is provided separately from the terminal device 8 and the robot 1, but the system is not limited to this configuration, and the control device 6 may be provided within the terminal device 8 or the robot 1. If the control device 6 is provided in the terminal device 8, the output from the operating device 83 may be input to the 3D position identification unit 63 without going through the transmitting / receiving unit 81 and the transmitting / receiving unit 61. If the control device 6 is provided within the robot 1, the transmitting / receiving unit 5 may function as the transmitting / receiving unit 61. Also, in Figure 1, the robot control device 7 is provided separately from the robot 1, but the system is not limited to this configuration, and the robot control device 7 may be provided within the robot 1. If the robot control device 7 is installed inside the robot 1, the control commands output from the robot control device 7 may be input to the drive control unit 3 without going through the transmitting / receiving unit 5.

[0033] Next, the operation of this embodiment will be described. Figure 4 is a flowchart showing an example of the processing procedure in the control device 6 of this embodiment. As shown in Figure 4, the control device 6 determines whether or not a target position has been specified (step S1). In detail, the 3D position identification unit 63 determines whether or not it has received the target position from the transmitting / receiving unit 61. As described above, the operator specifies the position of the target point to be reached as the target position in the camera image by operating the terminal device 8. When the terminal device 8 receives input of the target position from the operator, it transmits the target position to the control device 6. Note that Figure 4 shows an example in which the viewpoint of the initial state of the shooting device 2 is used as the first viewpoint, but the first viewpoint is not limited to the viewpoint of the initial state of the shooting device 2, i.e., the initial shooting position and shooting direction. If the first viewpoint is not the viewpoint of the initial state of the shooting device 2, the viewpoint is changed based on the operator's specification before step S1. For example, if the shooting direction is linked to the shooting position, the viewpoint is determined when the shooting position is determined, so the viewpoint is changed when the shooting position is specified. If the shooting position and shooting direction can be set separately, the viewpoint is specified by the operator when the shooting position and shooting direction are specified by the operator.

[0034] If no target position is specified (Step S1 No), the control device 6 repeats Step S1. If a target position is specified (Step S1 Yes), the control device 6 stores it as the target position in the first viewpoint (Step S2). Specifically, the 3D position identification unit 63 stores the target position received from the transmitting / receiving unit 61 as the target position in the first viewpoint (hereinafter also referred to as the first target position).

[0035] Next, the control device 6 determines whether or not a viewpoint change instruction has been received (step S3). Specifically, the 3D position identification unit 63 determines whether or not a viewpoint change instruction has been received from the transmitting / receiving unit 61. As described above, the operator inputs a viewpoint change instruction to the terminal device 8 by operating the terminal device 8. When the terminal device 8 receives the viewpoint change instruction input from the operator, it transmits the viewpoint change instruction to the control device 6.

[0036] If a viewpoint change instruction is received (Step S3 Yes), the control device 6 instructs the camera to change its viewpoint (Step S4). Specifically, in Step S4, the 3D position identification unit 63 transmits the viewpoint change instruction received from the transmitting / receiving unit 61 to the robot 1 via the transmitting / receiving unit 61.

[0037] Next, the control device 6 determines whether a target position has been specified (step S5). Specifically, the 3D position identification unit 63 determines whether it has received the target position from the transmitting / receiving unit 61, similar to step S1. If no target position has been specified (step S5 No), the control device 6 repeats the process from step S3. If a target position has been specified (step S5 Yes), the control device 6 stores it as the target position in the second viewpoint (step S6). Specifically, the 3D position identification unit 63 stores the target position received from the transmitting / receiving unit 61 as the target position in the second viewpoint (hereinafter also referred to as the second target position). As described above, the first target position and the second target position are the same target point, but the first target position indicates the position in the camera image captured in the first viewpoint, and the second target position indicates the position in the camera image captured in the second viewpoint.

[0038] Next, the control device 6 calculates the three-dimensional position of the target destination (step S7). Specifically, the 3D position identification unit 63 uses the first target position, the second target position, and the camera model stored in the model storage unit 62 to calculate the three-dimensional position of the target destination, and outputs the calculated three-dimensional position to the transmitting / receiving unit 61.

[0039] Figure 5 is a diagram illustrating the method for calculating the three-dimensional position of the target destination in this embodiment. Figure 5 is a view of the monocular camera 21 from above, showing an example where the target destination of the robot 1 is the position of target 300. The components of the robot 1 other than the monocular camera 21 are not shown. The monocular camera 21 in the first viewpoint acquires image 301 by taking pictures with direction 400 as the shooting direction, and the monocular camera 21 in the second viewpoint acquires image 302 by taking pictures with direction 401 as the shooting direction. In the example shown in Figure 5, the monocular camera 21 moves from the position corresponding to the first viewpoint to the position corresponding to the second viewpoint by rotating in the rotation direction 303. Note that the images 301 and 302 shown in Figure 5 do not actually indicate that the images are placed in the positions shown, but rather schematically show that images corresponding to the field of view of the monocular camera 21 are acquired.

[0040] The operator, while reviewing the video 301 acquired from the first viewpoint, uses the terminal device 8 to designate the position corresponding to the target 300 in the video 301 as the target position (first target position). In the example shown in Figure 5, from the first viewpoint, the target 300 is located to the right of direction 400, so in video 301, it will also be located slightly to the right of the left-right center within video 301. Similarly, from the second viewpoint, the target 300 is located to the right of direction 401, so in video 302, it will also be located slightly to the right of the left-right center within video 302.

[0041] The 3D position identification unit 63 uses the camera model, the first target position specified in the video 301, and the second target position specified in the video 302 to calculate the target direction 402, which is the direction of the target 300 at the position corresponding to the first viewpoint, and the target direction 403, which is the direction of the target 300 at the position corresponding to the second viewpoint. The 3D position identification unit 63 uses the camera model to calculate the position corresponding to the first viewpoint and the position corresponding to the second viewpoint, and uses the position corresponding to the first viewpoint, the position corresponding to the second viewpoint, the target direction 402, and the target direction 403 to calculate the three-dimensional position of the target 300, i.e., the three-dimensional position of the target destination, using the principle of triangulation. The rotation angle from the position corresponding to the first viewpoint to the position corresponding to the second viewpoint can be, for example, 90 degrees, but is not limited to this. Note that Figure 5 is an example, and the direction and amount of movement of the monocular camera 21 from the first viewpoint to the second viewpoint are not limited to the example shown in Figure 5.

[0042] Returning to the explanation of Figure 4, after step S7, the control device 6 outputs the calculated 3D position (step S8) and terminates the process. In step S8, more specifically, the transmitting / receiving unit 61 transmits the 3D position received from the 3D position identification unit 63 to the robot control device 7.

[0043] If the answer in step S3 is No, the control device 6 determines whether a target position has been specified, similar to step S1 (step S9). If a target position has been specified (step S9 Yes), the control device 6 proceeds to step S6. If a target position has not been specified (step S9 No), the control device 6 repeats the process from step S3.

[0044] Through the above process, the 3D position of the target point is transmitted to the robot control device 7, and the robot control device 7 can operate the robot 1 based on the 3D position of the target point. In this embodiment, the operator does not need to perform stereoscopic vision, but can view the camera image displayed on the terminal device 8 as a 2D image and specify the target position in each of the two camera images with different viewpoints. This reduces the mental load on the operator. In addition, it is difficult to perceive depth, i.e., the direction of shooting, with a single camera image, but in this embodiment, the operator can view two camera images with different viewpoints, so the operator can also perceive depth.

[0045] In the example described above, the viewpoint was changed by changing the position of the monocular camera 21. However, camera images corresponding to multiple viewpoints captured by the monocular camera 21 may be stored in a storage unit (not shown) of the control device 6, or in a device other than the control device 6. When predetermined conditions are met, the system may switch to processing using multiple camera images without driving the monocular camera 21. It may also be possible to manually switch between displaying the actual image and the stored image. Furthermore, the actual image and the stored image may be displayed with an overlap. For example, the transparency of the stored image may be increased to create an overlap between the actual image and the stored image. The method of displaying overlaps is not limited to this example. Processing using multiple camera images is a process in which the control device 6 generates a 3D image using the multiple stored camera images. The predetermined conditions may include, for example, the condition that a predetermined number of camera images have been stored, or the condition that the number of viewpoints corresponding to the stored camera images has reached a predetermined number. However, the system is not limited to these conditions, and any condition that enables the generation of a 3D image is acceptable. The control device 6 transmits the generated 3D image to the terminal device 8, causing the display device 82 of the terminal device 8 to display the 3D image. This presents the 3D image to the operator, who can perform operations such as specifying the target position while viewing the 3D image. For example, the 3D image is displayed on a 3D monitor capable of stereoscopic display, and the operator specifies the target position in the 3D image using the operating means while sequentially changing the viewpoint. Alternatively, the control device 6 may perform a process using multiple camera images, generating a display image corresponding to a first viewpoint specified by the operator and an image corresponding to a second viewpoint specified by the operator, using free-viewpoint video technology with the stored camera images. The terminal device 8 displays these images and accepts the specification of the target position within each of these images, thereby performing the same process as described in Figure 4.By processing images from multiple cameras, power consumption can be reduced compared to operating the camera drive mechanism 23 each time, and a sense of security can be given to people around the robot 1 (the anxiety caused by the operation of the camera drive mechanism 23 can be suppressed).

[0046] In the example shown in Figure 1, one monocular camera 21 is used, but multiple monocular cameras 21 may be used. Figure 6 shows an example of the configuration of the remote control system of this embodiment when multiple monocular cameras 21 are used. The remote control system 100a shown in Figure 6 is the same as the remote control system 100 shown in Figure 1, except that it includes robot 1a instead of robot 1. Robot 1a is the same as robot 1 shown in Figure 1, except that it includes imaging device 2a instead of imaging device 2. Components having the same function as in the example shown in Figure 1 are given the same reference numerals as in Figure 1, and redundant explanations are omitted.

[0047] In the example shown in Figure 6, the imaging device 2a is equipped with multiple monocular cameras 21. In Figure 6, the imaging device 2a is equipped with two monocular cameras 21, but there may be three or more monocular cameras 21. Each of the multiple monocular cameras 21 is installed in a different position on the robot 1a. The position of the monocular camera 21 on the robot 1a may be fixed, or it may be changeable by the camera drive mechanism 23, as in the example shown in Figure 1. In the example shown in Figure 6, the terminal device 8 displays camera images captured by one of the multiple monocular cameras 21, and the operator specifies the target position in the displayed camera image. Switching between the monocular cameras 21 to be used may be done automatically, for example, by automatically changing the monocular camera 21 to be used in response to a change in viewpoint by operation of the terminal device 8, or it may be done by operation by the operator. For example, the operator may specify the first target position and the second target position by intentionally specifying and switching the monocular camera 21 to be used to specify the target position. Alternatively, the terminal device 8 may display multiple camera images captured by the two monocular cameras 21, and the operator may input the target positions in the two camera images as the first target position and the second target position, respectively.

[0048] Furthermore, although a monocular camera 21 was used as the camera in the example shown in Figure 1, a 3D camera, also known as a depth camera or depth-sensing camera, may be used instead of the monocular camera 21. Figure 7 shows an example of the configuration of the remote control system of this embodiment when a 3D camera is used. The remote control system 100b shown in Figure 7 is the same as the remote control system 100 shown in Figure 1, except that it includes robot 1b instead of robot 1. Robot 1b is the same as robot 1 shown in Figure 1, except that it includes imaging device 2b instead of imaging device 2. Components having the same function as in the example shown in Figure 1 are given the same reference numerals as in Figure 1, and redundant explanations are omitted.

[0049] In the example shown in Figure 7, the imaging device 2b is equipped with a 3D camera 24 instead of a monocular camera 21. In the example shown in Figure 7, the terminal device 8 also displays depth information as part of the camera image. The terminal device 8 may display depth information, for example, by changing the color according to the depth, or it may use 3D data using depth information to display it so that the operator can perceive depth in response to movements for instructing the operator to change their viewpoint. For example, when the operator moves a part of their body such as their head, or moves the terminal device 8 that the operator is operating, the terminal device 8 may generate and display a 2D image similar to the one when the viewpoint is changed, based on 3D data using depth information. This image may be generated by the camera control unit 22, or it may be done by control in the terminal device 8 (not shown). Note that although a control device 6 is shown in Figure 7, if the 3D camera 24 has the function of a control device 6, the remote control system 100b does not need to have a control device 6. If the 3D camera 24 also functions as a control device 6, then the 3D camera 24 will have the functions of both an imaging device capable of acquiring three-dimensional information of the object being photographed and a control device 6.

[0050] When using the 3D camera 24, the 3D camera 24 acquires images from a fixed point, and the range in which depth can be detected is the visible range of the 3D camera 24. If the 3D camera 24 is not moved and images are acquired from a fixed point, the camera drive mechanism 23 does not need to be provided. On the other hand, there may be cases where the operator wants to check depth beyond the visible range of the 3D camera 24, and the camera drive mechanism 23 may be provided to allow the position of the 3D camera 24 to be changed. For example, if the camera control unit 22 determines that the amount of viewpoint change that exceeds the visible range of the 3D camera 24 has been specified by the operator based on a viewpoint change instruction received from the terminal device 8 via the transmitting / receiving unit 5, the camera drive mechanism 23 may be controlled to change the position of the 3D camera 24 according to the amount of viewpoint change. Note that if the 3D camera 24 starts moving after exceeding the visible range of the 3D camera 24, a delay will occur for the duration of the 3D camera 24's movement. To avoid this delay, the camera control unit 22 may start the movement of the 3D camera 24 by controlling the camera drive mechanism 23 before exceeding the visible range. For example, the camera control unit 22 may start moving the 3D camera 24 by controlling the camera drive mechanism 23 when the cumulative value of the viewpoint change amount exceeds a certain amount. Alternatively, the camera control unit 22 may control the camera drive mechanism 23 so that when a change in viewpoint is instructed, the 3D camera 24 starts moving in the instructed direction.

[0051] Furthermore, the camera control unit 22 or the terminal device 8 may generate the image displayed on the terminal device 8 so that, regardless of whether the 3D camera 24 is moving or not, the image displayed on the terminal device 8 is a two-dimensional image assuming that the viewpoint has moved according to the amount of change instructed by the operator.

[0052] Next, the hardware configuration of the control device 6 of this embodiment will be described. In the control device 6 of this embodiment shown in Figures 1, 6, and 7, the computer system functions as the control device 6 when a program, which is a computer program describing the processing in the control device 6, is executed on the computer system. Figure 8 is a diagram showing an example of the configuration of a computer system that realizes the control device 6 of this embodiment. As shown in Figure 8, this computer system comprises a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 constitute a processing circuit.

[0053] In Figure 8, the control unit 101 is a processor such as a CPU (Central Processing Unit), which executes a program describing the processing in the control device 6 of this embodiment. Note that a part of the control unit 101 may be implemented by dedicated hardware such as a GPU (Graphics Processing Unit) or FPGA (Field-Programmable Gate Array). The input unit 102 is a button, keyboard, mouse, touchpad, etc. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory), as well as storage devices such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained during processing, etc. The storage unit 103 is also used as a temporary storage area for the program. The display unit 104 is a display, such as an LCD (Liquid Crystal Display). Note that the display unit 104 and the input unit 102 may be integrated and implemented as a touch panel, etc. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker, etc. Note that Figure 8 is just one example, and the configuration of the computer system is not limited to the example shown in Figure 8. For example, the computer system that implements the control device 6 does not necessarily have to include an input unit 102, a display unit 104, and an output unit 106.

[0054] Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system with the above configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). When the program is executed, the program read from the storage unit 103 is stored in the main memory area of ​​the storage unit 103. In this state, the control unit 101 performs processing as the control device 6 of this embodiment according to the program stored in the storage unit 103.

[0055] In the above explanation, a program describing the processing in the control device 6 is provided using a CD-ROM or DVD-ROM as the recording medium. However, the explanation is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, a program provided via a transmission medium such as the Internet may be used.

[0056] The 3D position identification unit 63 shown in Figures 1, 6, and 7 is realized by executing a computer program stored in the storage unit 103 shown in Figure 8 by the control unit 101 shown in Figure 8. The storage unit 103 shown in Figure 8 is also used to realize the 3D position identification unit 63 shown in Figures 1, 6, and 7. The model storage unit 62 shown in Figures 1, 6, and 7 is part of the storage unit 103 shown in Figure 8. The transmitting and receiving unit 61 shown in Figures 1, 6, and 7 is realized by the communication unit 105 shown in Figure 8. Note that the control device 6 may be realized by multiple computer systems. Also, for example, the control device 6 may be realized by a cloud computer system.

[0057] Furthermore, the terminal device 8 shown in Figures 1, 6, and 7 is also realized by, for example, the computer system shown in Figure 8. The display device 82 shown in Figures 1, 6, and 7 is realized by the display unit 104 shown in Figure 8. The operating device 83 shown in Figures 1, 6, and 7 is realized by the input unit 102 shown in Figure 8. The transmitting and receiving unit 81 shown in Figures 1, 6, and 7 is realized by the communication unit 105 shown in Figure 8.

[0058] Furthermore, the robot control device 7 shown in Figures 1, 6, and 7 can also be realized, for example, by the computer system shown in Figure 8. In addition, at least a part of the camera control unit 22 and drive control unit 3 shown in Figures 1, 6, and 7 may be realized by a processing circuit comprising the control unit 101 and storage unit 103 shown in Figure 8.

[0059] As described above, in this embodiment, the camera mounted on the robot 1 displays camera images taken from a first viewpoint and a second viewpoint on the terminal device 8. The terminal device 8 receives input of the target position, which is the position of the target point to be reached in each camera image, and uses the received target position to identify the three-dimensional position of the target point to be reached. This reduces the brain load on the operator while allowing the operator to perceive depth. Furthermore, since the motion parallax of the monocular camera 21 is utilized, only a display device 82 that displays two-dimensional images is needed, eliminating the need for dedicated hardware for stereoscopic viewing such as a head-mounted display or lenticular. In addition, because the method specifies the target point of the robot 1, remote operation is possible even in the presence of communication delays. Moreover, by applying this embodiment, it is possible to shorten working time, reduce operator fatigue, and improve work accuracy.

[0060] Embodiment 2. Figure 9 shows an example of the configuration of a remote control system according to Embodiment 2. The remote control system 100c of this embodiment includes a terminal device 8a usable by an operator remotely controlling the robot 1, a control device 6a for identifying the three-dimensional position of the target destination of the robot 1, and a shooting device 2. Components having the same functions as in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and redundant descriptions are omitted. The following mainly describes the differences from Embodiment 1.

[0061] Terminal device 8a is the same as terminal device 8 in Embodiment 1, except that a sensor 84 for detecting the operator's movements is added. Sensor 84 may be an accelerometer or gyro sensor that detects the acceleration, tilt, etc., of terminal device 8a as the operator's movements, or it may be a camera that acquires video footage of the operator to detect the movements of the operator's eyes, head, etc. Sensor 84 only needs to be capable of detecting the operator's movements and is not limited to the examples described above. In this embodiment, the operator changes the viewpoint of the monocular camera 21 by their own movements. That is, the viewpoint of the monocular camera 21 is switched by the operator's movements. For example, similar to Embodiment 1, the operator specifies a target position in the first viewpoint and then changes the viewpoint of the monocular camera 21 by turning their head to the right. In the configuration example shown in Figure 2, for example, when the operator turns their head to the right, the viewpoint of the monocular camera 21 rotates by a predetermined angle by the camera drive mechanism 23. The correspondence between the operator's movements and how the viewpoint of the monocular camera 21 is changed may be predetermined, for example, but may be changeable by the operator.

[0062] Movements to change the viewpoint of the monocular camera 21 may include, for example, movements using the terminal device 8, such as tilting the terminal device 8, rotating the terminal device 8, or shaking the terminal device 8. Alternatively, movements to change the viewpoint of the monocular camera 21 may include, for example, changing the operator's line of sight, moving the operator's hand, changing the direction of the operator's face by turning their head, or moving the position of the operator's head. The movements to change the viewpoint of the monocular camera 21 are not limited to these.

[0063] In Figure 9, the sensor 84 is located inside the terminal device 8a, but if the sensor 84 is a camera that acquires the operator's image, the sensor 84 may be located outside the terminal device 8a. The sensor information, which is the detection result of the sensor 84, is transmitted to the control device 6a by the transmitting / receiving unit 81.

[0064] The control device 6a is the same as the control device 6 of Embodiment 1, except that a conversion unit 64 is added. When the transmitting / receiving unit 61 receives sensor information from the terminal device 8a, it outputs the received sensor information to the conversion unit 64. The conversion unit 64 converts the movement detected by the sensor 84 into a change in the viewpoint of the imaging device 2. For example, the conversion unit 64 converts the operator's movement into a viewpoint change instruction based on the sensor information, and outputs the viewpoint change instruction obtained by the conversion to the transmitting / receiving unit 61 and the 3D position identification unit 63. For example, the conversion unit 64 holds information indicating the correspondence between the operator's movement and how to change the viewpoint of the monocular camera 21 as movement correspondence information, and uses the movement correspondence information to convert the operator's movement detected from the sensor information into a viewpoint change instruction. The transmitting / receiving unit 61 transmits the viewpoint change instruction received from the conversion unit 64 to the robot 1.

[0065] The 3D position identification unit 63 uses the target position received from the transmitting / receiving unit 61, the viewpoint change instruction received from the conversion unit 64 (i.e., the amount of viewpoint change converted by the conversion unit 64), and the camera model stored in the model storage unit 62 to calculate the three-dimensional position of the target destination, similar to Embodiment 1. The operation of this embodiment, other than that described above, is the same as in Embodiment 1.

[0066] Figure 9 shows an example in which the conversion unit 64 is provided within the control device 6a, but the configuration is not limited to this, and the conversion unit 64 may also be provided within the terminal device 8a. In this embodiment as well, the control device 6a may be provided within the terminal device 8a, within the robot control device 7, or within the robot 1. The control device 6a and terminal device 8a in this embodiment are also implemented, for example, by the computer system shown in Figure 8, similar to the control device 6 and terminal device 8.

[0067] Furthermore, when multiple monocular cameras 21 are used, as shown in Figure 6 of Embodiment 1, the terminal device 8a and control device 6a of this embodiment may be used to switch viewpoints in response to the operator's movements. Also, when multiple 3D cameras 24 are used, as shown in Figure 7 of Embodiment 1, the terminal device 8a and control device 6a of this embodiment may be used to switch viewpoints in response to the operator's movements.

[0068] As described above, in this embodiment, the viewpoint of the monocular camera 21 is switched in conjunction with the operator's movements, thereby improving the operator's sense of realism in depth perception.

[0069] Embodiment 3. Figure 10 shows an example of the configuration of a remote control system according to Embodiment 3. The remote control system 100d of this embodiment includes a terminal device 8, a control device 6, and a camera device 2, similar to Embodiment 1. In this embodiment, the camera device 2 is mounted on a robot 1c. The robot 1c is the same as the robot 1 of Embodiment 1, except that a collision sensor 10 for detecting collisions is added. The collision sensor 10 is, for example, a sensor that detects the possibility of a collision occurring. Components having the same functions as in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and redundant descriptions are omitted. The differences from Embodiment 1 will be mainly described below.

[0070] The robot 1c of this embodiment, similar to the robot 1 shown in Figure 2, for example, is equipped with a robot arm 42 having an end effector 43, and a collision sensor 10 is attached to the end effector 43. The collision sensor 10 detects whether there is a possibility of collision by determining whether the distance between it and surrounding objects, walls, people, etc. falls below a certain value. For example, when the collision sensor 10 detects that there is a possibility of collision, it may notify the drive control unit 3 of the detection result that there is a possibility of collision, or it may notify the drive control unit 3 of the distance between it and surrounding objects, walls, people, etc. as a detection result. The drive control unit 3 uses the detection result of the collision sensor 10 to detect a collision of the robot 1c and stops the operation of the robot 1c. For example, when the drive control unit 3 is notified by the collision sensor 10 that there is a possibility of collision, it may stop the operation of the robot 1c. Alternatively, when the drive control unit 3 is notified of the distance between it and surrounding objects, walls, people, etc., it may stop the operation of the robot 1c if the distance falls below a certain value. This prevents the robot 1c from colliding with surrounding objects, walls, people, etc.

[0071] In the above example, a collision sensor 10 was added to robot 1 in the configuration example shown in Figure 1, but the configuration is not limited to this. A collision sensor 10 could also be added to robot 1a shown in Figure 6, or robot 1b shown in Figure 7, and similarly, the drive control unit 3 could stop the operation of robots 1a and 1b when it is notified by the collision sensor 10 that a collision is possible. In the second embodiment, a collision sensor 10 is added to robot 1, and similarly, the drive control unit 3 could stop the operation of robot 1 when it is notified by the collision sensor 10 that a collision is possible.

[0072] Embodiment 4. Figure 11 shows an example of a display screen according to Embodiment 4. The configuration of the remote control system 100 in this embodiment is the same as that of the remote control system 100 shown in Figure 1 of Embodiment 1. Components having the same functions as in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and redundant descriptions are omitted. The differences from Embodiment 1 will be mainly described below.

[0073] In this embodiment, the robot 1 is equipped with a robot arm 42 having an end-effector 43, as illustrated in Figure 2. In this embodiment, when the end-effector 43 reaches the target point, the camera image displayed on the terminal device 8 is automatically magnified. The upper part of Figure 11 shows the camera image displayed on the display device 82 of the terminal device 8 before magnification. When the end-effector 43 reaches the target point, the camera image displayed on the display device 82 of the terminal device 8 is magnified, as shown in the lower part of Figure 11. This allows the operator to easily perform a gripping operation with the end-effector 43.

[0074] For example, when robot 1 reaches the target point, the drive control unit 3 notifies the camera control unit 22 that the target point has been reached. Upon receiving this notification, the camera control unit 22 enlarges the camera image and transmits the enlarged camera image to the terminal device 8 via the transmitting / receiving unit 5. Alternatively, upon receiving this notification, the camera control unit 22 may instruct the terminal device 8 via the transmitting / receiving unit 5 to enlarge and display the camera image, and the display control unit of the terminal device 8 (not shown) may control the display to enlarge the camera image. The instruction to enlarge the camera image may also be given via the control device 6.

[0075] In addition, the camera images displayed on the terminal devices 8, 8a may be enlarged in any of the remote control systems 100a shown in Figure 6, 100b shown in Figure 7, 100c described in Embodiment 2, and 100d described in Embodiment 3, or in any combination of two or more of these.

[0076] The configurations shown in the above embodiments are merely examples, and it is possible to combine them with other known technologies, combine different embodiments, and omit or modify parts of the configuration without departing from the gist of the invention.

[0077] The various aspects of this disclosure are summarized below as an appendix.

[0078] (Note 1) A camera capable of capturing images of an area including the target point reached by a remote machine, and capable of changing the viewpoint by changing the shooting position, A display device that displays the image captured by the aforementioned camera, An operating device that can receive a specification of the viewpoint of the shooting device from the operator, and can receive a specification of the target position, which is the position of the target arrival point of the remote machine in the video, A control device that calculates the three-dimensional position of the target point using a first target position, which is the target position in the first video, as specified by the operator based on a first video captured by the camera with a first viewpoint designated as the viewpoint; a second target position, which is the target position in the second video, as specified by the operator based on a second video captured by the camera with a second viewpoint designated as the viewpoint; and the shooting position of the camera corresponding to the second viewpoint. A remote control system characterized by having the following features. (Note 2) The aforementioned imaging device is A monocular camera, A camera drive mechanism that can change the shooting position of the monocular camera, The remote control system according to Appendix 1, characterized by comprising the following: (Note 3) The aforementioned imaging device is Multiple monocular cameras, The remote control system according to Appendix 1, characterized by comprising the following: (Note 4) The aforementioned imaging device is 3D camera, The remote control system according to Appendix 1, characterized by comprising the following: (Note 5) The aforementioned imaging device is A camera drive mechanism capable of changing the shooting position of the three-dimensional camera, The remote control system according to Appendix 4, characterized by comprising the following: (Note 6) A sensor that detects the operator's movements, A conversion unit that converts the motion detected by the sensor into a change in the viewpoint of the shooting device, Equipped with, The shooting device changes the viewpoint based on the amount of change converted by the conversion unit. The remote control system according to any one of Appendix 1 to 5, characterized in that the control device calculates the shooting position of the shooting device corresponding to the first viewpoint and the shooting position of the shooting device corresponding to the second viewpoint using the amount of change converted by the conversion unit. (Note 7) Equipped with collision sensors to detect collisions, The remote control system according to any one of the appendices 1 to 6, characterized in that the remote machine stops operating when it detects a collision using the detection result of the collision sensor. (Note 8) The remote control system according to any one of the appendices 1 to 7, characterized in that the image displayed on the display device is enlarged when the remote machine reaches the target point. (Note 9) The remote control system according to any one of the appendices 1 to 8, characterized in that the imaging device is mounted on the remote machine. (Note 10) The remote-controlled machine comprises a rotatable trolley and a robotic arm that can rotate around a fixed axis coaxial with the rotatable axis of the trolley. The remote control system according to Appendix 9, characterized in that the camera in the shooting device is fixed to the trolley and moves in conjunction with the rotation of the trolley. (Note 11) A remote control method in a remote control system equipped with a camera capable of photographing an area including the target point reached by a remote machine, and capable of changing the viewpoint by changing the shooting position, The steps include: displaying the image captured by the aforementioned camera; The process includes receiving input of a first target position, which is the position of the target destination within the first video, as specified by the operator, based on the first video captured by the camera while the first viewpoint is designated as the viewpoint; The process includes receiving input of a second target position, which is the position of the target destination in the second video, as specified by the operator, based on the second video captured by the camera while the second viewpoint is designated as the viewpoint, A step of calculating the three-dimensional position of the target destination using the first target position, the second target position, the shooting position of the shooting device corresponding to the first viewpoint, and the shooting position of the shooting device corresponding to the second viewpoint. A remote control method characterized by including the following. [Explanation of Symbols]

[0079] 1,1a,1b,1c Robot, 2,2a,2b Imaging device, 3 Drive control unit, 4 Drive mechanism, 5,61,81 Transmitter / receiver unit, 6,6a Control device, 7 Robot control device, 8,8a Terminal device, 9 Object, 10 Collision sensor, 21 Monocular camera, 22 Camera control unit, 23 Camera drive mechanism, 24 3D camera, 41 Cart, 42 Robot arm, 43 End effector, 62 Model memory unit, 63 3D position identification unit, 64 Conversion unit, 82 Display device, 83 Operating device, 84 Sensor, 100,100a,100b,100c,100d Remote control system.

Claims

1. A camera capable of capturing images of an area including the target point reached by a remotely controlled machine, A display device that displays the image captured by the aforementioned camera, An operating device that can receive a specification of the viewpoint of the shooting device from the operator, and can receive a specification of a target position from the operator, which is the position in the image that represents the target point reached by the machine in the image, A control device that calculates the three-dimensional position of the target destination using a first target position, which is the target position in the first video, as specified by the operator based on a first video captured by the camera with a first viewpoint designated as the viewpoint; a second target position, which is the target position in the second video, as specified by the operator based on a second video, as captured by the camera with a second viewpoint designated as the viewpoint; and the shooting position of the camera corresponding to the second viewpoint. A sensor that detects the operator's movements, A conversion unit that converts the motion detected by the sensor into a change in the viewpoint of the shooting device, Equipped with, The aforementioned imaging device is A monocular camera, A camera drive mechanism that can change the viewpoint by changing the shooting position of the monocular camera, Equipped with, The shooting device changes the viewpoint based on the amount of change converted by the conversion unit. The shooting position of the shooting device corresponding to the first viewpoint is predetermined or calculated based on the first viewpoint received by the operating device. The control device is characterized in that it calculates the shooting position of the shooting device corresponding to the second viewpoint using the shooting position of the shooting device corresponding to the first viewpoint and the amount of change obtained by the conversion unit from the movement detected by the sensor after the first target position has been specified.

2. A camera capable of capturing images of an area including the target point reached by a remotely controlled machine, A display device that displays the image captured by the aforementioned camera, An operating device that can receive a specification of the viewpoint of the shooting device from the operator, and can receive a specification of a target position from the operator, which is the position in the image that represents the target point reached by the machine in the image, A control device that calculates the three-dimensional position of the target destination using a first target position, which is the target position in the first video, as specified by the operator based on a first video captured by the camera with a first viewpoint designated as the viewpoint; a second target position, which is the target position in the second video, as specified by the operator based on a second video, as captured by the camera with a second viewpoint designated as the viewpoint; and the shooting position of the camera corresponding to the second viewpoint. A sensor that detects the operator's movements, A conversion unit that converts the motion detected by the sensor into a change in the viewpoint of the shooting device, Equipped with, The aforementioned imaging device is Multiple monocular cameras at different shooting positions, Equipped with, The shooting device changes the viewpoint based on the amount of change converted by the conversion unit. The shooting position of the shooting device corresponding to the first viewpoint is predetermined or calculated based on the first viewpoint received by the operating device. The control device is characterized in that it calculates the shooting position of the shooting device corresponding to the second viewpoint using the shooting position of the shooting device corresponding to the first viewpoint and the amount of change obtained by the conversion unit from the movement detected by the sensor after the first target position has been specified.

3. Equipped with collision sensors to detect collisions, The remote control system according to claim 1, characterized in that the machine stops operating when it detects a collision using the detection result of the collision sensor.

4. The remote control system according to claim 1, characterized in that the image displayed on the display device is enlarged when the machine reaches the target point.

5. The remote control system according to claim 1, characterized in that the imaging device is mounted on the machine.

6. The machine comprises a rotatable trolley and a robotic arm that can rotate around a fixed axis coaxial with the rotatable axis of the trolley. The remote control system according to claim 5, characterized in that the camera in the shooting device is fixed to the trolley and moves in conjunction with the rotation of the trolley.

7. A camera capable of capturing images of an area including the target point reached by a remotely controlled machine, A display device that displays the image captured by the aforementioned camera, An operating device that can receive a specification of the viewpoint of the shooting device from the operator, and can receive a specification of a target position from the operator, which is the position in the image that represents the target point reached by the machine in the image, A control device that calculates the three-dimensional position of the target destination using a first target position, which is the target position in the first video, as specified by the operator based on a first video captured by the camera with a first viewpoint designated as the viewpoint; a second target position, which is the target position in the second video, as specified by the operator based on a second video, as captured by the camera with a second viewpoint designated as the viewpoint; and the shooting position of the camera corresponding to the second viewpoint. Equipped with, The shooting position of the shooting device corresponding to the first viewpoint is predetermined or calculated based on the first viewpoint received by the operating device. The shooting device changes the viewpoint based on the amount of change in viewpoint from the first viewpoint received by the operating device. The control device calculates the shooting position of the shooting device corresponding to the second viewpoint using the shooting position of the shooting device corresponding to the first viewpoint and the change amount received by the operating device. The aforementioned imaging device is A monocular camera, A camera drive mechanism that can change the viewpoint by changing the shooting position of the monocular camera, Equipped with, When the control device satisfies the specified conditions, it generates a display image to be displayed on the display device using the multiple stored images. The display device displays the image shown, The remote control system is characterized in that the operating device accepts the designation of a position in the displayed image representing the target destination of the machine.

8. A remote control method in a remote control system comprising a shooting device capable of photographing an area including the target destination of a remotely controlled machine, equipped with multiple monocular cameras at different shooting positions and capable of changing the viewpoint by changing the shooting position, and a sensor that detects the movement of the operator, The steps include: displaying the image captured by the aforementioned camera; The process includes receiving input of a first target position, which is the position of the target destination within the first video, as specified by the operator, based on the first video captured by the camera while the first viewpoint is designated as the viewpoint; The process includes receiving input of a second target position, which is the position of the target destination within the second video, as specified by the operator, based on the second video captured by the camera while the second viewpoint is designated as the viewpoint, The steps include: converting the motion detected by the sensor into the amount of change in the viewpoint of the shooting device; A step of calculating the shooting position of the shooting device corresponding to the second viewpoint using the shooting position of the shooting device corresponding to the first viewpoint and the amount of change obtained by converting the movement detected by the sensor after the first target position has been specified, A step of calculating the three-dimensional position of the target destination using the first target position, the second target position, the shooting position of the shooting device corresponding to the first viewpoint, and the shooting position of the shooting device corresponding to the second viewpoint. Includes, A remote control method characterized in that the shooting position of the shooting device corresponding to the first viewpoint is predetermined or calculated based on the first viewpoint specified by the operator.

Images