Remote control system and remote control method

Abstract

A remote control system (100) comprises: a robot (1) provided with a sensor (12); a transparent touch panel (31) that receives, as an operation with respect to the robot (1), an action of an operator for causing the robot (1) to perform the action; a camera (13) that captures a camera image including the surroundings of the robot (1); a sensor image generation unit (23) that uses a detection value detected by the sensor (12) to generate a sensor image corresponding to the detection value; an operation detection unit (21) that detects an operation position that is a position in the camera image corresponding to the operation; a composite image generation unit (24) that determines, on the basis of the operation position, a position at which the sensor image is to be superimposed on the camera image and generates a composite image by superimposing the sensor image on the camera image at the determined position; and a display (33) that displays the composite image.

Description

Remote control system and remote control method

[0001] The present disclosure relates to a remote control system and a remote control method for remotely controlling edge devices.

[0002] In a remote control system that remotely controls edge devices such as robots, it is necessary to communicate the status of the edge devices to an operator. Patent Document 1 discloses a remote control system in which a robot is equipped with an image sensor, a tactile sensor, etc., and an operator wears a head-mounted display and a tactile presentation device. The remote control system described in Patent Document 1 transmits to the operator images captured by a camera mounted on the edge device and tactile sensations detected by the tactile sensor, allowing the operator to perform intuitive operations.

[0003] International Publication No. 2019 / 225548

[0004] The technology described in Patent Document 1 requires the use of a tactile presentation device that transmits tactile sensations detected by a tactile sensor mounted on the robot to the operator using physical force, which makes the device used by the operator more complex.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a remote control system that allows an operator to perform intuitive operations without complicating the device used by the operator.

[0006] In order to solve the above-mentioned problems and achieve the objectives, the remote control system disclosed herein comprises an edge device equipped with a sensor, an operation device that accepts an operator's actions to operate the edge device as operations on the edge device, a camera that captures camera footage that is an image including the surroundings of the edge device, a sensor image generation unit that uses the detection value detected by the sensor to generate sensor footage that is an image corresponding to the detection value, an operation detection unit that detects an operation position that is a position in the camera image that corresponds to the operation, a composite image generation unit that determines a position at which to superimpose the sensor footage on the camera footage based on the operation position, and generates a composite image by superimposing the sensor footage on the camera footage at the determined position, and a display device that displays the composite image.

[0007] According to the present disclosure, an effect is achieved in that an operator can perform an intuitive operation without complicating the device used by the operator.

[0008] FIG. 1 is a diagram showing an example of the configuration of a remote control system according to the first embodiment. FIG. 1 is a diagram showing an example of the appearance of a robot and a terminal device according to the first embodiment. FIG. 1 is a diagram showing an example of the operation of a hand according to the first embodiment. FIG. 1 is a diagram showing an example of the attachment position of a sensor according to the first embodiment. FIG. 2 is a flowchart showing an example of the operation in a control device according to the first embodiment. FIG. 2 is a diagram showing a specific example of a composite image according to the first embodiment. FIG. 3 is a diagram showing a specific example of a composite image according to the first embodiment. FIG. 4 is a diagram showing a specific example of a composite image according to the first embodiment. FIG. 1 shows an example of an operation method for independently operating each finger of the hand according to the first embodiment. FIG. 2 shows an example of an operation method for operating two fingers of the hand according to the first embodiment. FIG. 3 shows an example of an operation by a gesture according to the first embodiment. FIG. 4 shows an example of a composite image when an operation is performed by a gesture according to the first embodiment. FIG. 5 shows an example of the configuration of a computer system for realizing a control device according to the first embodiment. FIG. 6 shows another example of the configuration of a remote control system according to the second embodiment. FIG. 7 shows an example of a sensor image according to the second embodiment when sensors of the same type are provided in one location. FIG. 8 shows an example of a sensor image according to the second embodiment when sensors of different types are provided in one location. FIG. 9 shows an example of the configuration of a remote control system according to the third embodiment. FIG. 10 shows an example of a composite image according to the fourth embodiment. FIG. 11 shows another example of a composite image according to the fourth embodiment.

[0009] Hereinafter, a remote control system and a remote control method according to an embodiment will be described in detail with reference to the accompanying drawings.

[0010] First Embodiment. FIG. 1 is a diagram illustrating an example of a configuration of a remote control system according to a first embodiment. The remote control system 100 of this embodiment includes a robot 1 as an example of an edge device, a terminal device 3 as an example of an operation device that accepts operation inputs for remotely controlling the robot 1, and a control device 2 that generates an image to be displayed on the terminal device 3. The edge device is a machine (a machine at a remote location) equipped with a drive mechanism that can be remotely controlled by an operator operating the operation device in the remote control system 100. The remote control system 100 is also referred to as a mixed haptics remote control system, for example, but the name of the remote control system 100 is not limited to this. In the following, an example will be mainly described in which the edge device is a robot 1 that combines a cart and a robot arm (manipulator). However, the edge device may be any remotely controlled device equipped with a drive mechanism, such as a mobile vehicle, a robot arm, a humanoid robot, or an industrial or household machine, but is not limited to these.

[0011] The robot 1 includes a drive mechanism 11, a sensor 12 that detects at least one of the robot 1 and its surroundings, and a camera 13 that captures video of the robot 1's surroundings. The sensor 12 includes, for example, at least one of a force-tactile sensor and a temperature sensor. While the following description primarily focuses on an example in which the sensor 12 is a force-tactile sensor, the sensor 12 is not limited thereto. For example, as described above, the sensor 12 may be a temperature sensor, a pressure sensor, an acceleration sensor, or any other sensor that detects something other than video. The sensor 12 outputs sensor information indicating the detection results to the control device 2. The sensor information is output, for example, periodically, but is not limited thereto. The sensor 12 may also output the sensor information upon receiving an instruction from the control device 2. Although FIG. 1 illustrates two sensors 12, the number of sensors 12 may be one or more, and the number of sensors 12 is not limited to the example shown in FIG. 1.

[0012] The drive mechanism 11 includes actuators that drive the movable parts of the robot 1, and operates the robot 1 based on drive commands received from the control device 2. The movable parts include at least one of a moving mechanism for movement, such as tires or crawlers, and a working mechanism for performing work, such as a robot arm. The drive mechanism 11 may be integrated with the movable parts. The camera 13 outputs camera images, which are images obtained by shooting, to the control device 2.

[0013] In this embodiment, the control device 2 converts sensor information into an image, generates a composite image by superimposing the image on the camera image, and transmits the composite image to the terminal device 3. The terminal device 3 displays the composite image, allowing the operator to perform operations while checking the composite image. This allows the operator to perform intuitive operations without complicating the device used by the operator. For example, the haptic sensation detected by a force-tactile sensor can be transmitted to the operator without using a haptic presentation device that transmits physical force to the operator.

[0014] On the other hand, when generating a composite image, the control device 2 needs to superimpose the sensor image, which is an image showing sensor information, at an appropriate position. However, analyzing the camera image to identify the location for superimposition increases the computational load. For example, if the sensor 12 is provided at the hand of the robot 1, it is desirable to superimpose the sensor image near the hand of the robot 1, so computation is required to identify the hand from the camera image. Furthermore, if there is an obstruction between the hand of the robot 1 and the object being operated by the robot 1, an estimation process is required to identify the location for superimposing the sensor image, which increases the computational load. In this embodiment, the control device 2 determines the location for superimposing the sensor image on the camera image based on the position specified by the operator using the terminal device 3 for operating the robot 1. This reduces the computational load for identifying the location for superimposing the sensor image while maintaining the operator's intuitive operability.

[0015] The terminal device 3 includes a transparent contact panel 31, a communication unit 32, and a display 33. The display 33 is an example of a display device that displays images such as camera images and composite images. The transparent contact panel 31 is an example of an operation device that accepts an operator's actions for operating the robot 1 as operations for the robot 1 and acquires operation information indicating the accepted operations. The transparent contact panel 31 is integrated with the display 33 and is, for example, attached to the display 33. This allows a position on the display 33 to correspond to a position on the transparent contact panel 31, allowing the operator to specify a desired position in the image displayed on the display 33 via the transparent contact panel 31. The terminal device 3 is, for example, a smartphone, a tablet, a personal computer with a touch panel, or the like. The terminal device 3 may include a camera.

[0016] The communication unit 32 transmits and receives information by communicating with other devices. The communication unit 32 communicates with, for example, the control device 2. The communication unit 32 receives, for example, composite images, camera images, etc. (described later) from the control device 2 and outputs them to the display 33. The display 33 displays the composite images, camera images, etc. received from the communication unit 32. The communication unit 32 receives, from the transparent touch panel 31, operation information input by the operator when the operator touches the transparent touch panel 31, and transmits the received operation information to the control device 2. The operation information includes, for example, position information indicating the operated position (position on the display 33), i.e., the position where the operator's finger touched the transparent touch panel 31. The operation information may also include information indicating the strength of pressure applied to the transparent touch panel 31.

[0017] The control device 2 includes an operation detection unit 21, a drive command generation unit 22, a sensor image generation unit 23, a composite image generation unit 24, and a communication unit 25. The communication unit 25 transmits and receives information by communicating with other devices. The communication unit 25 communicates with, for example, the terminal device 3. For example, the communication unit 25 receives operation information from the terminal device 3 that the terminal device 3 has accepted from an operator, and outputs the received operation information to the operation detection unit 21. The communication unit 25 also transmits the composite image received from the composite image generation unit 24 to the terminal device 3.

[0018] The operation detection unit 21 detects an operation position, which is a position in the camera image corresponding to the operation, from the input operation information. Specifically, the operation detection unit 21 detects the operation position and an operation instruction (operation instruction to the robot 1) from the operation information received from the communication unit 25, outputs the operation instruction to the drive command generation unit 22, and if the operation instruction is an operation instruction for an operation to superimpose a sensor image, outputs the operation position to the composite image generation unit 24. The operation detection unit 21 determines, for example, a position in the camera image corresponding to a position indicated by the position information as the operation position. Specifically, for example, if a position on the transparent touch panel 31 is expressed by coordinate values ​​in a coordinate system fixed to the display 33, the coordinate values ​​on the display 33 correspond to the coordinate values ​​in the camera image. In this case, the position indicated by the position information in the operation information directly corresponds to the position in the camera image. The operation instruction may be, for example, an instruction to move the robot 1, an instruction to operate a manipulator of the robot 1, or an instruction to operate a hand (finger) of the manipulator, but is not limited thereto, and may be an instruction to operate a movable part of the robot 1. The action on which the sensor image is superimposed (hereinafter also referred to as the action to be superimposed) is at least a part of all the action instructions corresponding to the robot 1, and is, for example, but not limited to, the action of the moving part corresponding to the sensor 12. The action to be superimposed will be described later.

[0019] The sensor image generator 23 generates a sensor image, which is an image corresponding to the detection value detected by the sensor 12, using the detection value. Specifically, the sensor image generator 23 converts the sensor information received from the sensor 12 into a sensor image and outputs the sensor image to the composite image generator 24. The sensor image will be described in detail later, but the sensor image is an image whose color, shape, hue, etc. change according to the detection value of the sensor 12.

[0020] The composite image generation unit 24 determines a position at which to superimpose the sensor image on the camera image based on the operation position, and generates a composite image by superimposing the sensor image on the camera image at the determined position. Specifically, the composite image generation unit 24 receives the camera image from the camera 13, and based on the operation position received from the operation detection unit 21, superimposes the sensor image received from the sensor image generation unit 23 on the camera image to generate a composite image by combining the camera image and the sensor image, and outputs the composite image to the communication unit 25. For example, the composite image generation unit 24 superimposes the sensor image on the camera image so that the sensor image is superimposed at a position including the operation position. Note that if the sensor image is not superimposed, i.e., if the operation position is not received from the operation detection unit 21, the composite image generation unit 24 outputs the camera image to the communication unit 25.

[0021] The drive command generation unit 22 generates a drive command for operating the robot 1 based on the operation instruction received from the operation detection unit 21, and outputs the generated drive command to the drive mechanism 11 of the robot 1. The drive command includes, for example, information indicating the moving part to be operated and a command value indicating the amount by which the moving part is to be driven. Note that, when there is only one moving part, the drive command does not need to include information indicating the moving part to be operated.

[0022] Although Figure 1 shows an example in which information is exchanged between each part of the robot 1 and each part of the control device 2, the robot 1 may be equipped with a communication unit, and the communication unit of the robot 1 may communicate with the communication unit 25 of the control device 2 or a communication unit other than the communication unit 25, thereby transmitting information between each part of the robot 1 and each part of the control device 2.

[0023] When the robot 1 and the terminal device 3 are located apart, the control device 2 may be provided in an area where the robot 1 is installed, in an area where the terminal device 3 is located, or in an area different from both the robot 1 and the terminal device 3. For example, the control device 2 may be realized by a cloud computer system. The control device 2 may be incorporated into the robot 1 and integrated therewith, or may be incorporated into the terminal device 3 and integrated therewith. As will be described later, the components constituting the control device 2 may be provided separately in the robot and the terminal device. In addition, although the drive command generation unit 22 is provided in the control device 2 in FIG. 1 , a drive control device including the drive command generation unit 22 may be provided separately from the control device 2, and the drive command generation unit 22 may not be provided in the control device 2.

[0024] FIG. 2 is a diagram showing an example of the appearance of the robot 1 and the terminal device 3 according to this embodiment. In the example shown in FIG. 2, the robot 1 includes a robot arm 14 and a movable dolly 16. The robot arm 14 includes a hand 15 capable of grasping a target 4, which is an object of operation by the robot arm 14. Camera images captured by the camera 13 of the robot 1 are displayed on the display 33 of the terminal device 3. The dolly 16 includes movable parts, such as wheels and crawlers, that are driven by actuators such as motors. As described above, the transparent contact panel 31 is attached to the display 33, and the operator can view the camera images displayed on the display 33 through the transparent contact panel 31.

[0025] 2, the operator can input an operation instruction to move the cart 16 to the terminal device 3 by operating the transparent contact panel 31, and can also input an operation instruction to move the hand 15 to the terminal device 3 by operating the transparent contact panel 31. In the example shown in Fig. 2, the robot arm 14 moves in conjunction with an operation instruction to the hand 15, so an operation instruction to the hand 15 also corresponds to an instruction to the robot arm 14. The operation of the robot 1 is not limited to this, and the operation of the entire robot arm 14 and the operation of the hand 15 may be controlled separately.

[0026] In the example shown in FIG. 2 , the operation is performed with the operator's fingers. Here, it is assumed that the rules for remotely controlling the robot 1 stipulate that the operator moves one finger on the transparent contact panel 31 to move the cart 16, and that the operator moves two fingers on the transparent contact panel 31 to grasp the target 4 with the hand 15. For example, to move the cart 16, the operator touches one finger on the transparent contact panel 31 and pulls the finger toward himself. The driving force of the cart 16 may be increased by pulling the finger toward himself more strongly, or the driving force of the cart 16 may be increased by increasing the speed at which the finger is pulled toward himself. In FIG. 2 , the operator is attempting to use two fingers to cause the hand 15 of the robot 1 to grasp the target 4. Note that the rules for remotely controlling the robot 1 that correspond to the actions of the robot 1 and the actions of the operator are not limited to the above example, and any rules may be used.

[0027] FIG. 3 is a diagram illustrating an example of the operation of the hand 15 according to the present embodiment. As shown on the left side of FIG. 3 , the operator touches the transparent touch panel 31 by touching two fingers to the transparent touch panel 31 and moves the two fingers closer to each other on the transparent touch panel 31. As a result, the hand 15 of the robot 1 approaches the target 4, as shown on the right side of FIG. 3 . Specifically, the operation detection unit 21 of the control device 2 determines that two fingers have touched the transparent touch panel 31 based on the position information included in the operation information, since there are two contact points. Based on this determination result, the operation detection unit 21 detects that the operation instruction is an instruction to move the hand 15. The operation detection unit 21 outputs the operation position along with the operation instruction to the actuation command generation unit 22. Alternatively, the operation detection unit 21 may determine the movement amount of the hand 15 based on a change in the operation position over time and output the movement amount along with the operation instruction to the actuation command generation unit 22. The actuation command generation unit 22 determines the movement amount of the hand 15 based on a change in the operation position over time, generates an operation command corresponding to the operation instruction, and operates the robot 1. In this way, the robot 1 operates in conjunction with the movement of the operator's fingers. As shown in Figures 2 and 3, the camera image is displayed on the display 33 of the terminal device 3, so the operator can operate the robot 1 while checking the position of the hand 15 of the robot 1 and the target 4.

[0028] 2 and 3, a camera image without a sensor image superimposed thereon is displayed on the display 33. On the other hand, when the operator brings the hand 15 into contact with the target 4, if the operator can recognize information related to haptics, the sense of realism increases and operability improves.

[0029] FIG. 4 is a diagram showing an example of the mounting position of the sensor 12 in this embodiment. In the example shown in FIG. 4, the sensor 12, which is a force-tactile sensor, is attached to the fingertips of the hand 15. Note that the robot 1 illustrated in FIG. 2 has two hands 15, and therefore the sensor 12 is attached to the fingertips of each hand 15. In this embodiment, for example, a force-tactile sensor is provided as the sensor 12 on the fingertips of the hand 15, and the control device 2 superimposes a sensor image on the camera image so that a sensor image is displayed on the display 33 near the finger that the operator touches on the transparent touch panel 31 to operate the hand 15. This gives the operator the illusion that the finger operating the hand 15 is linked to the sensor image, allowing the operator to operate the hand 15 while intuitively feeling the force and touch.

[0030] 5 is a flowchart showing an example of the operation of the control device 2 according to the present embodiment. As shown in FIG. 5, the control device 2 acquires camera images and sensor information (step S1). Specifically, the composite image generator 24 acquires camera images from the camera 13, and the sensor image generator 23 acquires sensor information from the sensor 12.

[0031] Next, the control device 2 determines whether or not an operation has been performed (step S2). As described above, the operator operates the robot 1 by touching the transparent contact panel 31 of the terminal device 3 with a finger. This causes the transparent contact panel 31 of the terminal device 3 to transmit operation information to the control device 2 via the communication unit 32. Therefore, in step S2, more specifically, the operation detection unit 21 determines whether or not operation information has been received from the terminal device 3 via the communication unit 25.

[0032] If no operation is performed (step S2: No), the control device 2 transmits the camera image (step S3) and repeats the process from step S1. Specifically, in step S3, the composite image generation unit 24 outputs the camera image to the communication unit 25, and the communication unit 25 transmits the camera image to the terminal device 3. As a result, the camera image is displayed on the display 33 of the terminal device 3.

[0033] If an operation has been performed (Yes in step S2), the control device 2 detects the operation position and the operation instruction from the operation information (step S4). Specifically, the operation detection unit 21 detects the operation position and the operation instruction from the operation information and outputs the operation instruction to the drive command generation unit 22.

[0034] The control device 2 generates a drive command based on the operation instruction (step S5) and outputs the drive command (step S6). In detail, the drive command generation unit 22 generates a drive command for the robot 1 based on the operation instruction received from the operation detection unit 21, and outputs the generated drive command to the drive mechanism 11 of the robot 1.

[0035] The control device 2 determines whether the instructed motion is a motion to be superimposed (step S7). Specifically, the operation detection unit 21 determines whether the detected motion instruction is a motion to be superimposed. Note that if all of the motions of the robot 1 are motions to be superimposed, step S7 does not need to be performed.

[0036] If the instructed action is not a superimposition target action (step S7: No), the control device 2 performs step S3 and repeats the process from step S1. If the instructed action is a superimposition target action (step S7: Yes), the control device 2 generates a sensor image (step S8). In detail, if the instructed action is a superimposition target action, the operation detection unit 21 instructs the sensor image generation unit 23 to generate a sensor image, and the sensor image generation unit 23 generates a sensor image based on the sensor information and outputs the generated sensor image to the composite image generation unit 24.

[0037] The control device 2 generates a composite image of the camera image and the sensor image (step S9). Specifically, the composite image generation unit 24 generates the composite image by superimposing the sensor image on the camera image based on the operation position received from the operation detection unit 21.

[0038] For example, the composite image generation unit 24 may superimpose the sensor image on the camera image so that the center of the sensor image is the operation position, or may superimpose the sensor image on the camera image so that the bottom edge of the sensor image is the operation position. Alternatively, the composite image generation unit 24 may superimpose the sensor image on the camera image so that the center of the sensor image is offset above the operation position, i.e., so that the center of the sensor image is shifted upward by the offset amount from the center of the sensor image. The offset amount may be, for example, predetermined or may be changeable by the operator. Because the sensor image is displayed near the operator's finger, the offset amount is set so that the position of the sensor image is shifted upward compared to when the sensor image is superimposed on the camera image so that the center of the sensor image is the operation position, so that the operator can easily sense that the finger and the sensor image are linked. However, it is desirable to set the offset amount so that the operation position is included in the area where the sensor image is superimposed when the sensor image is offset.

[0039] If the two hands 15 are each provided with a sensor 12, the sensor image generating unit 23 outputs a sensor image associated with, for example, the identification information of the sensor 12, and two operation positions are output from the operation detecting unit 21. For example, the relative positional relationship of each sensor 12 in the camera image is defined in advance based on the imaging range of the camera 13 and the configuration of the robot 1, and the composite image generating unit 24 determines which sensor image each operation position corresponds to based on this definition.

[0040] The control device 2 transmits the composite video (step S10) and repeats the process from step S1. In step S10, more specifically, the composite video generation unit 24 outputs the composite video to the communication unit 25, and the communication unit 25 transmits the composite video to the terminal device 3. As a result, the composite video is displayed on the display 33 of the terminal device 3.

[0041] Note that the procedure shown in Fig. 5 is an example, and the order of processing is not limited to the example shown in Fig. 5 as long as equivalent processing can be realized. Furthermore, in the example shown in Fig. 5, a sensor image is generated when the instructed action is a superimposition target action, but this is not limiting. The sensor image generation unit 23 may generate a sensor image each time it receives sensor information and output it to the composite image generation unit 24. In this case, even if the composite image generation unit 24 receives a sensor image, if it has not received an operation position from the operation detection unit 21, it does not superimpose the sensor image on the camera image.

[0042] Next, specific examples of composite images according to this embodiment are shown in Figures 6 to 11. Figures 6 to 11 show a state in which the operator has started an operation to grasp the target 4 illustrated in Figure 2, with the operator placing two fingers in contact with the transparent touch panel 31 of the terminal device 3.

[0043] In the example shown in FIG. 6 , when the operator brings two fingers closer together on the transparent contact panel 31 from the state shown on the left side of the figure, the two hands 15 of the robot 1 approach the target 4, as shown on the right side of FIG. 6 . Note that in the following figures, when the operator's fingers are shown with dashed lines, this indicates that the transparent contact panel 31 and the display 33 are shown with the operator's fingers virtually transparent. In the example shown in FIG. 6 , contact positions 201 and 202 where the operator's two fingers touch the transparent contact panel 31 are detected by the control device 2 as operation positions, and superimposed images 203 and 204, which are sensor images, are superimposed around positions offset above the operation positions. The superimposed images 203 and 204 are sensor images corresponding to the sensors 12 provided on the two hands 15, respectively. In the example shown in FIG. 6 , the high and low detection values ​​indicated by the sensor information are indicated by the shade of the color of the sensor images. In this way, the superimposed images 203 and 204 are displayed near the operator's finger based on the position where the operator touches the transparent touch panel 31 with their finger for operation, giving the operator the illusion that the color shade of the finger is changing. In other words, the operator is given the illusion that they are feeling a haptic sensation due to the change in the color shade of the finger. In this way, the superimposed images 203 and 204 are superimposed in conjunction with the operator's operation, allowing the operator to perform operations while intuitively feeling a haptic sensation. Note that, although the level of the detection value indicated by the sensor information is indicated by the color shade of the sensor image in FIG. 6 , the color, i.e., the hue, may be changed depending on the level of the detection value indicated by the sensor information.

[0044] 7, the level of the detection value indicated by the sensor information is expressed by the number of rings in the superimposed images 203 and 204. For example, in the example shown in Fig. 7, the sensor images are generated so that the number of rings increases as the detection value indicated by the sensor information increases. The overall size of the superimposed images 203 and 204 may be increased by adding rings to the outside as the detection value indicated by the sensor information increases, or the density of the rings may be increased as the detection value indicated by the sensor information increases by changing the number of rings without changing the display size of the superimposed images 203 and 204.

[0045] In the example shown in Fig. 8, the high and low detection values ​​indicated by the sensor information are expressed by different graphic shapes in the superimposed images 203 and 204. For example, when the detection value indicated by the sensor information is higher or lower than the state shown in Fig. 8, the superimposed images 203 and 204 are shown in a different graphic shape from that shown in Fig. 8, such as a circle or a square. Note that the graphic shapes are not limited to these examples and may be any graphic shape.

[0046] 9, the high and low detection values ​​indicated by the sensor information are expressed by differences in the movement of the images, such as vibration and trembling, in the superimposed images 203 and 204. For example, the frequency of the vibration, the pattern of time change of the vibration, etc. are changed depending on the high and low detection values ​​indicated by the sensor information.

[0047] 10, the high or low of the detection value indicated by the sensor information is expressed by the difference in the blinking state of the image in the superimposed images 203 and 204. For example, the frequency of the blinking, the pattern of the blinking over time, etc. are changed depending on the high or low of the detection value indicated by the sensor information.

[0048] 11, the superimposed image 205, which is a sensor image, includes text. In the example shown in Fig. 11, text corresponding to the detection value indicated by the sensor information is determined in advance, and for example, onomatopoeia such as "soft and fluffy," "squish," and "gyuu!" are determined according to the detection value, and the onomatopoeia corresponding to the detection value are displayed as text in the superimposed image 205. Note that the text to be displayed is not limited to these, and it is sufficient that text corresponding to the detection value indicated by the sensor information is displayed.

[0049] The sensor image is not limited to the examples shown in FIGS. 6 to 11 . For example, the size of the image may be changed according to the detection value indicated by the sensor information, or the position at which the image is displayed may be changed according to the detection value indicated by the sensor information. For example, the sensor image may be generated so that a horizontally elongated figure, like an indicator, is displayed higher the higher the detection value indicated by the sensor information. The sensor image may also be a combination of the above examples. For example, both the shape and color of the sensor image may change according to the detection value indicated by the sensor information. The display method of the sensor image may also be changeable by an operator.

[0050] Furthermore, although the above description has mainly focused on the case where the sensor 12 is a force-tactile sensor, a sensor image corresponding to the detected value indicated by the sensor information is similarly generated when the sensor 12 is a temperature sensor, pressure sensor, etc. As shown in Fig. 11, when indicating the temperature using text, onomatopoeia such as "warm" or "hot" may be used when the sensor 12 is a temperature sensor.

[0051] Next, an example of a screen that accepts operations from the operator will be described. As described above, the operator performs operations by touching the transparent touch panel 31 with a finger while the camera image is displayed on the display 33 of the terminal device 3. If the operator touches the finger at a position away from the portion corresponding to the sensor 12 of the robot 1, the sensor image may be displayed in an inappropriate position. For example, suppose the action to be superimposed is the operation of the hand 15, i.e., a grasping-related operation for grasping the target 4 with the hand 15, and the superimposed sensor image corresponds to the sensor 12 attached to the fingertips of the hand 15. The operator operates the hand 15 by touching two fingers at positions away from the hand 15. In this case, it is difficult for the operator to recognize the link between the operation of the hand 15 and the movement of the hand 15 in the image. Furthermore, the sensor image is displayed at a position away from the hand 15, making it difficult to recognize that the haptic sensation is detected by the hand 15. For this reason, an operable range in which the operation of the superimposed target action can be accepted may be defined, and if contact with the transparent contact panel 31 is detected within the operable range, it may be accepted as an operation, and if contact with the transparent contact panel 31 is detected outside the operable range, it may not be accepted as an operation.

[0052] FIG. 12 is a diagram illustrating an example of the operable range according to the present embodiment. In the example illustrated in FIG. 12, the operable range is displayed by a dashed frame 206. The composite image generation unit 24 may superimpose a frame indicating the operable range on the camera image and transmit the camera image with the superimposed frame to the terminal device 3 via the communication unit 25. The frame is not limited to a dashed line, and may be a solid line, a dotted line, or another line. Furthermore, the method of indicating the operable range is not limited to the method of indicating the operable range by a frame. The composite image generation unit 24 may indicate the operable range by processing either the inside or outside of the operable range of the camera image. For example, the composite image generation unit 24 may indicate the operable range by processing the camera image so as to blur the camera image outside the operable range, or by processing the camera image so as to reduce the color contrast of the outside of the operable range, or by processing the camera image so as to display the operable range as a color image and the outside of the operable range as a black and white image. The method of indicating the operable range is not limited to the above-mentioned example, and any method may be used.

[0053] Furthermore, as described above, because the operator performs operations with his or her finger, the sensor image may be obscured by the finger, making it difficult to see. For this reason, the example described above is one in which the sensor image is superimposed on the camera image so that the center of the sensor image is above the operation position indicated by the operator's finger. When the sensor image is superimposed by shifting it from the operation position, i.e., offsetting it from the operation position, if the offset amount is large, the operator may find it difficult to perceive the sensor image as being linked to the movement of the finger. For this reason, a graphic for linking the operation position and the sensor image (hereinafter also referred to as a linking graphic) may be displayed to make it easier for the operator to perceive the sensor image as being linked to the movement of the finger.

[0054] FIG. 13 is a diagram showing an example of an interlocking graphic according to this embodiment. In the example shown in FIG. 13, a graphic 207 is displayed as an interlocking graphic from the operation position where the operator's finger touches the transparent touch panel 31 upward. The graphic 207 is composed of a circle indicating the operation position, a bar-shaped graphic above the circle, and a circle above the bar-shaped graphic, and the sensor image is displayed in the upper circle portion. In the example shown in FIG. 13, the composite image generation unit 24 superimposes the graphic 207 from the operation position upward and also superimposes the sensor image in the circle portion above the graphic 207. Because the graphic 207 moves in accordance with the movement of the operator's finger, the operator can sense that the sensor image and the operation of the hand 15 are interlocked with the movement of the finger.

[0055] FIG. 14 is a diagram showing another example of an interlocking graphic according to the present embodiment. In the example shown in FIG. 14 , a graphic 208 is displayed as an interlocking graphic, extending upward from the operation position where the operator's finger touches the transparent touch panel 31. The graphic 208 has a shape that gradually becomes thinner upward from the operation position where the operator's finger touches the transparent touch panel 31, and a sensor image is superimposed on the tip of the graphic 208. The sensor image may be displayed so that the tip of the graphic 208 is at the center, or so that the tip of the graphic 208 is at the bottom. In this case, the graphic 208 also moves in accordance with the movement of the operator's finger, allowing the operator to feel that the sensor image and the operation of the hand 15 are interlocked with the movement of the finger. FIGS. 13 and 14 are merely examples, and the shape of the interlocking graphic is not limited to these examples.

[0056] Furthermore, the sensor image may be made more visible by using an attachment that allows a position away from the operator's finger to be specified as the operation position on the transparent contact panel 31. FIG. 15 is a diagram showing an example of operation using the attachment of this embodiment. In the example shown in FIG. 15, the operator wears the attachment 50 on his / her fingertip, so that the operation position is higher than the finger. As shown in FIG. 15, the attachment 50 is composed of, for example, a cap portion worn on the finger, a thin rod-shaped portion, and a tip portion that contacts the transparent contact panel 31. Note that the shape of the attachment 50 is not limited to the example shown in FIG. 15. By performing an operation by contacting the attachment 50 worn on the operator's finger with the transparent contact panel 31, it is possible to prevent the sensor image from being obscured by the operator's finger, even when the sensor image is superimposed on the camera image centered around the operation position. This makes the sensor image more visible.

[0057] Next, a description will be given of another example of a method for operating the robot 1. In Fig. 2, the two robot arms 14 are assumed to move in conjunction with each other by moving the hands 15 of the two robot arms 14 closer to or farther apart, so the operator operates the hands 15 with two fingers of one hand. However, this is not limiting, and each of the two robot arms 14 may be operated independently.

[0058] 16 is a diagram showing an example of an operation method for independently operating two robot arms 14 according to this embodiment. In the example shown in Fig. 16, the hands 15 of the two robot arms 14 are operated with the fingers of the right hand and the fingers of the left hand, respectively. In this case, similarly, superimposed images 301 and 302, which are sensor images corresponding to the sensors 12 attached to the hands 15 of the robot arms 14, are superimposed on the camera image based on the operation positions of the fingers.

[0059] Furthermore, the hand 15 of the robot arm 14 may have multiple fingers, and the multiple fingers of the hand 15 may be operable. In this case, a sensor 12 may be attached to each of the multiple fingers of the hand 15. FIG. 17 is a diagram showing an example of an operation method for operating two fingers of the hand 15 according to this embodiment. In the example shown in FIG. 17 , the two fingers of the hand 15 are operated with two fingers of one hand, similar to the operation of the hand 15 in the example shown in FIG. 2 , and superimposed images 303 and 304, which are sensor images, are superimposed on the camera image according to the operation position. Furthermore, if the hand 15 has three or more fingers, for example, the third and subsequent fingers of the hand 15 move in conjunction with one of the two fingers being operated.

[0060] Furthermore, in the above example, the operation device is a transparent touch panel 31. However, the superimposition of the sensor image in this embodiment is not limited to this example and can also be applied to, for example, an example in which an operator's gesture is accepted as an operation. FIG. 18 is a diagram showing an example of an operation by gesture in this embodiment. In this case, for example, the operator's operation is associated with the movement of the robot 1 so that the fingers of the hand 15 of the robot 1 move in the same manner as the fingers of the operator's hand. In the example shown in FIG. 18, the terminal device 3 includes a camera 34 as an operation device. With the camera image displayed on the terminal device 3, the camera 34 captures an image of the operator's hand, and the captured image is transmitted to the control device 2 as operation information via the communication unit 32. In this way, the operation device may be a device that captures an image of the operator. In this case, the operation information includes an image captured by the operation device. The operation detection unit 21 detects the operation position and the operation instruction from the image included in the operation information. Note that, although FIG. 18 shows a case in which the camera 34 is an external camera of the terminal device 3, the camera 34 may be a camera provided separately from the terminal device 3. Furthermore, if the operator's hand is between the operator and the terminal device 3, the camera 34 may be an in-camera provided in the terminal device 3.

[0061] In the example shown in FIG. 18 , the hand 15 has five fingers, each of which corresponds to one of the five fingers on the operator's hand. A sensor 12 is attached to the fingertip of each finger of the hand 15. The operation detection unit 21 analyzes the video captured by the camera 34, associates the movements of the operator's fingers with the movements of the fingers on the hand 15, and outputs the associated movements of the fingers of the hand 15 as operation instructions to the drive command generation unit 22. The operation detection unit 21 also detects a predetermined position (e.g., a fingertip) of the associated finger of the hand 15 as an operation position, and outputs the detected operation position to the composite image generation unit 24. Based on the received operation position, the composite image generation unit 24 superimposes a sensor image of the corresponding sensor 12 on the camera image for each operation position.

[0062] FIG. 19 is a diagram illustrating an example of a composite image when an operation is performed using a gesture according to this embodiment. In the example illustrated in FIG. 19 , the hand 15 has five fingers 305. As illustrated in FIG. 19 , to facilitate recognition that the sensor image is linked to the movement of the operator's fingers, the composite image generation unit 24 displays a hand image 306 representing the operator's hand in a portion of the camera image corresponding to the hand 15, with the background made transparent so that it can be seen through. The hand image 306 may be, for example, an image of the operator's hand extracted (cut out) from an image captured by the camera 34 in the camera image, or a CG (Computer Graphics) image simulating a human hand. It is preferable that the hand image 306 be displayed at a position corresponding to the hand 15. Therefore, for example, the composite image generation unit 24 may display the hand image 306 at a predetermined position in the camera image. If the hand image 306 is misaligned with the position of the hand 15, the display position of the hand image 306 may be changed based on the operator's operation. Alternatively, the composite image generation unit 24 may determine the position of the hand 15 from the camera image. In the example shown in FIG. 19, a sensor 12 is provided on each finger 305, and a sensor image 307 is displayed for each finger 305.

[0063] Furthermore, the number of fingers on the hand 15 is not limited to five. For example, if the hand 15 has three fingers, each finger on the hand 15 is previously associated with three of the operator's fingers. Similarly, when the movement of a moving part of the operator's body, such as the foot, arm, or head, is defined as a gesture for operation and the robot 1 operates in accordance with the gesture, the operation detection unit 21 similarly detects the defined position of the moving part as the operation position. That is, the operation detection unit 21 determines the operation position using the position of the moving part in the operation information.

[0064] Next, the operation for superimposing the sensor image will be described. In the example described above, the sensor image was superimposed while the operator's two fingers were in contact with the transparent touch panel 31. However, this is not limited to this. The sensor image may also be superimposed even when the operator's fingers are removed from the transparent touch panel 31 after the sensor image has been superimposed. In this case, the composite image generation unit 24, for example, considers the position where the operator's two fingers last touched the transparent touch panel 31 as the operation position and superimposes the sensor image on the camera image. Note that if the camera 13 is drivable and the shooting direction of the camera 13 can be changed, the composite image generation unit 24 acquires information indicating the drive angle of the camera 13 from the camera 13 and moves the operation position based on the drive angle of the camera 13, thereby superimposing the sensor image at a position corresponding to the operation position before the camera 13 was driven. For example, if the camera 13 is rotated 30 degrees, the operation position is moved to a position rotated -30 degrees in the camera image acquired after the rotation. Furthermore, when the terminal device 3 moves, a sensor provided in the terminal device 3 detects the movement of the terminal device 3, and the camera image displayed on the display 33 changes. The position of the sensor image superimposed on the camera image is also shifted by an amount corresponding to the change in the camera image. For example, if an operator moves the terminal device 3 while touching the transparent touch panel 31 at a position corresponding to the fingertip of the hand 15 displayed on the display 33, the position of the sensor image superimposed on the camera image shifts to match the fingertip of the hand 15. Note that if the movement range of the terminal device 3 exceeds the angle of view of the camera 13, the camera 13 is also driven in conjunction with the movement of the terminal device 3. Furthermore, if the composite image generation unit 24 determines that the camera image is busy, i.e., that there are many displayed objects, it may stop superimposing the sensor image. Whether the camera image is busy may be determined based on the camera image capacity, by analyzing the brightness distribution, or by other methods.

[0065] It is desirable to synchronize the movement of the robot 1 with the movement of the operator. That is, the actuation command generator 22 generates actuation commands according to operation information, thereby synchronizing the movement of the robot 1 with the movement of the operator. This improves the operability of the operator. For example, in the case of the robot 1 configured as shown in FIG. 2 , the movement of the robot 1 is synchronized with the movement of the operator by generating an actuation command so that the hand 15 of the robot 1 or the fingertips of the hand 15 move in synchronization with the movement of the operator's fingers.

[0066] The robot 1 may also be configured to automatically stop when it detects a collision. For example, the robot 1 may be equipped with a collision sensor, separate from the sensor 12, capable of detecting the distance to surrounding objects, structures, etc., and the robot 1 may stop its operation when it detects a collision of the robot 1 using the detection result of the collision sensor. Specifically, the robot 1 may be equipped with a stop determination unit that determines that a collision has occurred when the distance to a surrounding object, structure, etc., such as the hand 15 of the robot 1, becomes less than a threshold value, and automatically stops the robot 1. The drive command generation unit 22 of the control device 2 may also function as a stop determination unit, and may generate a drive command to stop the robot 1 when the distance to a structure, etc., becomes less than a threshold value, based on the detection result of the collision sensor of the robot 1.

[0067] Next, the hardware configuration of the control device 2 of this embodiment will be described. The control device 2 of this embodiment shown in FIG. 1 functions as the control device 2 by executing a computer program on the computer system, which is a computer program that describes the processing to be performed by the control device 2. FIG. 20 is a diagram showing an example configuration of a computer system that realizes the control device 2 of this embodiment. As shown in FIG. 20, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

[0068] In FIG. 20 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processing in the control device 2 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 includes buttons, a keyboard, a mouse, a touchpad, and the like. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is, for example, a display such as an LCD (Liquid Crystal Display). Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that Fig. 20 is just an example, and the configuration of the computer system is not limited to the example of Fig. 20. For example, the computer system that realizes the control device 2 does not need to include the input unit 102, the display unit 104, and the output unit 106.

[0069] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above configuration, for example, a computer program is installed into storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as control device 2 of this embodiment in accordance with the program stored in storage unit 103.

[0070] In the above explanation, a program describing the processing in the control device 2 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0071] The operation detection unit 21, drive command generation unit 22, sensor image generation unit 23, and composite image generation unit 24 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 20 executing a computer program stored in the storage unit 103 shown in FIG. 20. The storage unit 103 shown in FIG. 20 is also used to realize the operation detection unit 21, drive command generation unit 22, sensor image generation unit 23, and composite image generation unit 24 shown in FIG. 1. The communication unit 25 shown in FIG. 1 is realized by the communication unit 105 shown in FIG. 20. Note that the control device 2 may be realized by multiple computer systems. Furthermore, for example, the control device 2 may be realized by a cloud computer system.

[0072] Furthermore, when the terminal device 3 also functions as the control device 2, the terminal device 3 is also realized by, for example, the computer system shown in Fig. 20. In this case, the input unit 102 is the transparent contact panel 31 shown in Fig. 1, and the display unit 104 is the display 33. When the robot 1 also functions as the control device 2, the robot 1 is equipped with the computer system shown in Fig. 20. In this case, the robot 1 only needs to be equipped with a processing circuit including the control unit 101 and the memory unit 103 as a computer system, and may not be equipped with at least some of the other components.

[0073] As described above, the configuration example shown in FIG. 1 is merely an example, and the functions of the control device 2 may be distributed among multiple devices. FIG. 21 is a diagram showing another configuration example of a remote control system according to the present embodiment. Components having the same functions as those in FIG. 1 are designated by the same reference numerals as in FIG. 1, and redundant description will be omitted. The remote control system 100a shown in FIG. 21 includes a robot 1a and a terminal device 3a. The robot 1a is configured by adding a drive command generator 22, a sensor image generator 23, and a communication unit 25 to the robot 1 shown in FIG. 1. The terminal device 3a is configured by adding an operation detector 21 and a composite image generator 24 to the terminal device 3 shown in FIG. 1. In the remote control system 100a shown in FIG. 21, the functions of the control device 2 shown in FIG. 1 are distributed between the robot 1a and the terminal device 3a. In other words, the control device 2a shown in FIG. 21 is realized by multiple devices. Fig. 21 is an example, and the division of functions between the robot 1a and the terminal device 3a is not limited to the example shown in Fig. 21. For example, the robot 1a may be provided with the composite image generation unit 24. Furthermore, a composite image generation device equipped with the composite image generation unit 24 may be provided separately from the robot 1a and the terminal device 3a, and the functions of the control device 2 shown in Fig. 1 may be distributed among three devices.

[0074] As described above, in this embodiment, the remote control system 100, 100a displays a sensor image, which is an image showing a detection value acquired by the sensor 12, superimposed on a position (position in a camera image) specified by an operator operating the robot 1, 1a using a part of his or her body, such as a finger. Therefore, the device used by the operator only needs to include a display device for the operator to check the status of the robot 1, 1a and an operation device for accepting operation inputs. There is no need for an additional tactile presentation device or the like for the operator to recognize the detection value acquired by the sensor 12. The operator can perform operations while feeling as if the sensor image superimposed on the camera image is linked to a part of the operator's body. Therefore, this embodiment allows the operator to perform intuitive operations without complicating the device used by the operator. Furthermore, because the sensor image is superimposed on the camera image based on a position specified by operating the robot 1, 1a, calculations for determining the position at which to superimpose the sensor image are not required, thereby reducing the computational load.

[0075] Second Embodiment Fig. 22 is a diagram showing an example of the configuration of a remote control system according to a second embodiment. A remote control system 100b according to this embodiment includes a robot 1b, a control device 2b, and a terminal device 3. The terminal device 3 is similar to the terminal device 3 according to the first embodiment. Components having the same functions as those according to the first embodiment are given the same reference numerals as those according to the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0076] Similar to the first embodiment, the robot 1b includes a drive mechanism 11, a sensor 12 that detects at least one of the conditions of the robot 1b and the surroundings of the robot 1b, and a camera 13 that captures images of the surroundings. In the first embodiment, an example was described in which one sensor 12 is provided per part of the robot 1, i.e., per location, such as one sensor 12 provided on each of the two hands 15 or one sensor 12 provided on each finger of the hand 15. In the present embodiment, an example will be described in which multiple sensors 12 are provided at one location. Multiple sensors 12 of the same type may be provided at one location, or different types of sensors 12 may be provided. For example, multiple force-tactile sensors may be provided at slightly different positions as sensors 12 in one hand 15, or multiple force-tactile sensors and temperature sensors may be provided.

[0077] The control device 2b is similar to the control device 2 of the first embodiment except that it includes a sensor image generation unit 23a instead of the sensor image generation unit 23. The sensor image generation unit 23a generates a sensor image based on sensor information from a plurality of sensors 12 provided at one location, and outputs the generated sensor image to the composite image generation unit 24. As in the first embodiment, the composite image generation unit 24 superimposes the sensor image on the camera image based on the operation position.

[0078] FIG. 23 is a diagram illustrating an example of a sensor image according to the present embodiment when the same type of sensors 12 are installed in one location. As in the first embodiment, the sensor image is also superimposed on the camera image in the present embodiment. However, in FIG. 23, the sensor image portion is enlarged for clarity. In the example illustrated in FIG. 23, the sensor image generator 23a generates a contour diagram 400 as the sensor image based on the sensor information of the same type of sensors 12 and the installation positions of each sensor 12. The contour diagram 400 may be represented by shading or color coding. Note that the sensor image when the same type of sensors 12 are installed in one location is not limited to the example illustrated in FIG. 23; for example, the divided pressure may be represented by color coding or shading in a mesh pattern. In this way, the sensor image generator 23a may generate an image showing the distribution of detection values ​​as the sensor image using detection values ​​acquired by the same type of sensors 12.

[0079] FIG. 24 is a diagram illustrating an example of a sensor image according to this embodiment in which different types of sensors 12 are provided in one location. In FIG. 24 , the sensor image portion is enlarged for clarity. In the example illustrated in FIG. 24 , two types of sensors 12, such as a force-tactile sensor and a temperature sensor, are provided on the fingertips of one hand 15. In the example illustrated in FIG. 24 , the sensor image generator 23 a generates a sensor image including images 401 and 402 corresponding to the sensor information of the two sensors 12, respectively. For example, the images 401 and 402 are displayed in different colors, and the images 401 and 402 are displayed in shades corresponding to the detected values. The sensor images corresponding to the two types of sensors 12 are not limited to those illustrated in FIG. 24 . For example, the sensor image area may be divided into the number of sensors 12 corresponding to one location, and the detected values ​​of the sensor information of each sensor 12 may be represented for each divided area using the various sensor image representation methods described in the first embodiment. Furthermore, the representation methods corresponding to each sensor 12 do not need to be the same. For example, the detection value may be expressed in a different way for each sensor 12, such as by changing the shade depending on the detection value of the force-tactile sensor and by changing the hue depending on the detection value of the temperature sensor. Alternatively, instead of dividing the area for each sensor 12, an image may be generated in which the shade or hue is changed depending on the detection value of the force-tactile sensor as shown in Fig. 6 , and the detection value of the temperature sensor may be represented by the number of rings as shown in Fig. 7 , and the image showing the detection value of the temperature sensor may be superimposed on the image showing the detection value of the force-tactile sensor. In this way, the sensor image generator 23a may use detection values ​​acquired by a plurality of different types of sensors 12 to generate a sensor image including images that individually represent the detection values ​​of each type of sensor 12.

[0080] The control device 2b of this embodiment is realized, for example, by the computer system illustrated in FIG. 20 , similar to the control device 2 of embodiment 1. The control device 2b may be realized by multiple computers, for example, by a cloud computer system, similar to the control device 2 of embodiment 1. Furthermore, the control device 2b of this embodiment may be integrated with the robot 1b, similar to the control device 2b of embodiment 1, or may be integrated with the terminal device 3. Furthermore, the control device 2b of this embodiment may have its functions distributed across multiple devices, similar to the control device 2a of embodiment 1.

[0081] Third Embodiment Fig. 25 is a diagram showing an example of the configuration of a remote control system according to a third embodiment. A remote control system 100c according to this embodiment includes a robot 1c, a control device 2, and a terminal device 3. The control device 2 and the terminal device 3 are similar to the control device 2 and the terminal device 3 according to the first embodiment. Components having the same functions as those according to the first embodiment are given the same reference numerals as those according to the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0082] The robot 1c is similar to the robot 1 of the first embodiment except that a presentation device 17 is added to the robot 1 of the first embodiment. In this embodiment, the presentation device 17 presents sensor information acquired by the sensor 12 to those around it. The presentation device 17 is a device capable of displaying a detection value detected by the sensor 12. The presentation device 17 is, for example, an LED (Light Emitting Diode) or a small monitor, and emits light according to the detection value indicated by the sensor information, or displays at least one of text and an image according to the detection value. If the presentation device 17 is an LED, the detection value may be indicated by the lighting state of the LED (blinking, on, off, etc.), or different colored LEDs may be provided so as to emit light in different colors according to the detection value.

[0083] For example, if the sensor 12 is a force-tactile sensor, people around the robot 1c can know whether the hand 15 of the robot 1c is firmly or lightly gripping the target 4. This allows people around the robot 1c to monitor the gripping state of the target 4, or to take refuge if the target 4 is about to fall, thereby contributing to improved safety.

[0084] In this embodiment, the control device 2 is realized, for example, by the computer system illustrated in FIG. 20 , as in the first embodiment. The control device 2 may be realized by multiple computers, for example, by a cloud computer system, as in the control device 2 of the first embodiment. Furthermore, the control device 2 of this embodiment may be integrated with the robot 1 c, or may be integrated with the terminal device 3, as in the control device 2 of the first embodiment. Furthermore, the control device 2 of this embodiment may have its functions distributed among multiple devices, as in the control device 2 a of the first embodiment.

[0085] In addition, a presentation device 17 may be added to the robot 1b of the remote control system 100b of embodiment 2, and the presentation device 17 may emit light according to the detection value indicated by the sensor information or display according to the detection value.

[0086] 25 , a presentation device 17 is provided for each sensor 12, but a presentation device 17 may be provided for some of the multiple sensors 12. Furthermore, the presentation device 17 may include multiple types of LEDs, each of which may be associated with a respective sensor 12. Furthermore, if the presentation device 17 is a small monitor, the display area of ​​the small monitor may be divided, and the sensor information of the multiple sensors 12 may be displayed in each of the divided areas.

[0087] Fourth Embodiment. Figure 26 is a diagram showing an example of a composite image according to a fourth embodiment. The configuration of the remote control system 100 according to this embodiment is the same as that according to the first embodiment. In the first embodiment, an example has been described in which, when the hand 15 is operated in the configuration example shown in Figure 2, a sensor image corresponding to the sensor 12 attached to the hand 15 is superimposed on a camera image. The sensor image is not limited to this example, and may be, for example, an image showing the driving force that drives the movement mechanism. In this case, the sensor 12 is a sensor that detects the driving force, such as a motor current that indicates the driving force for moving the carriage 16.

[0088] As in the example described using Fig. 2, when moving the cart 16, the operator moves one finger on the transparent touch panel 31. In this embodiment, the movement of the cart 16 is included in the actions to be superimposed, and the operation detection unit 21 detects contact by one finger based on the operation information and determines that movement of the cart 16 has been instructed. The operation detection unit 21 detects the operation position based on the operation information and outputs the detected operation position to the composite image generation unit 24. The composite image generation unit 24 superimposes a sensor image corresponding to the sensor 12 that detects the driving force on the camera image.

[0089] Although only a portion of the dolly 16 is displayed on the display 33 in Fig. 26, the entire dolly 16 may be displayed as shown in Fig. 27. Fig. 27 is a diagram showing another example of the composite image according to the fourth embodiment. The image displayed in this case is, for example, an image captured by a separate camera attached to the top of the dolly 16, or CG using a known dolly model.

[0090] As shown in FIG. 26 , when the operator touches one finger to the transparent touch panel 31 to move the cart 16, an image 500, which is a sensor image corresponding to the sensor 12 detecting the driving force, is displayed superimposed on the camera image. This allows the operator to control the movement of the cart 16 while recognizing the driving force. For example, if the cart 16 does not move very far despite a large driving force, it is likely that the floor or road surface below the cart 16 is in poor condition, such as having many bumps and grooves. Conversely, if the cart 16 moves quickly despite a small driving force, it is likely that the floor or road surface below the cart 16 is smooth. In this way, by checking the driving force and the state of movement, the operator can control the movement of the cart 16 while understanding the condition of the floor or road surface. The driving force may be expressed using, but is not limited to, shading, hue, number of rings, flashing, vibration, or text, as described in the first embodiment.

[0091] Furthermore, the remote operation system 100 may perform both or only one of the superimposition of a sensor image showing the driving force when operating the carriage 16 to move and the superimposition of a sensor image showing the detection value of the sensor 12 when operating the hand 15 described in embodiment 1. Furthermore, the operator may be able to set whether to perform both or a selected one of these.

[0092] In the remote operation system 100b of the second embodiment or the remote operation system 100c of the third embodiment, a sensor image showing the driving force when operating the movement of the cart 16 may be similarly superimposed. In a remote operation system that combines the second and third embodiments, a sensor image showing the driving force when operating the movement of the cart 16 may be similarly superimposed.

[0093] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0094] Various aspects of the present disclosure are summarized below as appendices.

[0095] and a display device that displays the composite image. (Supplementary Note 1) A remote control system comprising: an edge device having a sensor; an operation device that accepts an operator's action to operate the edge device as an operation for the edge device; a camera that captures camera image that is an image including the periphery of the edge device; a sensor image generation unit that uses a detection value detected by the sensor to generate a sensor image that is an image corresponding to the detection value; an operation detection unit that detects an operation position that is a position in the camera image that corresponds to the operation; a composite image generation unit that determines a position at which to superimpose the sensor image on the camera image based on the operation position, and generates a composite image by superimposing the sensor image on the camera image at the determined position; and a display device that displays the composite image. (Supplementary Note 2) The remote control system according to Supplementary Note 1, wherein the operation is performed by the operator's finger; the operation device is a transparent touch panel integrated with the display device; operation information accepted by the operation device includes position information indicating a position at which the finger contacted the transparent touch panel during the operation; and the operation detection unit uses the position information to determine the position at which to superimpose the sensor image on the camera image. (Supplementary Note 3) The remote control system according to Supplementary Note 2, wherein the operation detection unit sets a position in the camera image corresponding to a position indicated by the position information as the operation position, and the synthetic image generation unit superimposes the sensor image on the camera image so that the sensor image is superimposed at a position including the operation position. (Supplementary Note 4) The remote control system according to Supplementary Note 3, wherein the operation detection unit sets a position indicated by the position information as the operation position, and the synthetic image generation unit superimposes the sensor image on the camera image so that the center of the sensor image is the operation position. (Supplementary Note 5) The remote control system according to Supplementary Note 3, wherein the operation detection unit sets a position indicated by the position information as the operation position, and the synthetic image generation unit superimposes the sensor image on the camera image so that the center of the sensor image is offset above the operation position.(Supplementary Note 6) The remote control system according to Supplementary Note 3, wherein the operation is performed by bringing an attachment worn on the operator's finger into contact with the transparent touch panel, and the synthetic image generation unit superimposes the sensor image on the camera image so that the center of the sensor image is the operation position. (Supplementary Note 7) The remote control system according to Supplementary Note 1, wherein the operation is performed by a moving part that is a part of the operator's body, the operation device includes a device that captures an image of the operator, the operation information received by the operation device includes an image captured by the operation device, and the operation detection unit determines the operation position using the position of the moving part in the operation information received by the operation device. (Supplementary Note 8) The remote control system according to any one of Supplements 1 to 7, wherein the edge device includes a hand capable of grasping a target, the sensor is attached to the hand, and the operation includes a grasping-related operation that is an operation for grasping the target with the hand. (Supplementary Note 9) The remote control system according to Supplementary Note 8, wherein the sensor includes a force-tactile sensor. (Supplementary Note 10) The remote control system according to Supplementary Note 8, wherein the sensor includes a temperature sensor. (Supplementary Note 11) The remote control system according to any one of Supplements 8 to 10, wherein the operation is performed by the operator's finger, the operation device is a transparent touch panel integrated with the display device, the operation information received by the operation device includes position information indicating a position where the finger touches the transparent touch panel during the operation, and the operation detection unit determines that the operation by the operator is the grip-related operation when two fingers of the operator simultaneously touch the transparent touch panel, and determines a position at which to superimpose the sensor image on the camera image using the position information of the grip-related operation. (Supplementary Note 12) The remote control system according to any one of Supplements 1 to 11, wherein the edge device includes a movement mechanism for movement, and the sensor includes a sensor for detecting a driving force that drives the movement mechanism.(Supplementary Note 13) The remote control system according to any one of Supplements 1 to 12, further comprising: a drive command generation unit that generates a drive command for the edge device, wherein the drive command generation unit synchronizes operation of the edge device with operation of the operator by generating a drive command according to operation information received by the operation device. (Supplementary Note 14) The remote control system according to any one of Supplements 1 to 13, wherein the synthetic image generation unit uses detection values ​​acquired by a plurality of the same type of sensors to generate, as the sensor image, an image showing a distribution of the detection values. (Supplementary Note 15) The remote control system according to any one of Supplements 1 to 14, wherein the synthetic image generation unit uses detection values ​​acquired by a plurality of different types of sensors to generate the sensor image including an image individually showing the detection values ​​of the sensors for each type. (Supplementary Note 16) The remote control system according to any one of Supplements 1 to 15, wherein the edge device comprises a collision sensor for detecting a collision, and wherein the edge device stops operation when it detects a collision of the edge device using the detection result of the collision sensor. (Supplementary Note 17) The remote operation system according to any one of Supplements 1 to 16, wherein the edge device is equipped with a presentation device capable of displaying detection values ​​detected by the sensor. (Supplementary Note 18) A remote operation method in a remote operation system that remotely operates an edge device equipped with a sensor, comprising: a step of accepting an action of an operator for operating the edge device as an operation for the edge device, a step of capturing camera video that is video including the surroundings of the edge device, a step of generating a sensor video that is video corresponding to the detection values ​​detected by the sensor using the detection values, a step of detecting an operation position that is a position in the camera video that corresponds to the operation, a step of determining a position at which to superimpose the sensor video on the camera video based on the operation position, and generating a composite video by superimposing the sensor video on the camera video at the determined position, and a step of displaying the composite video.

[0096] 1, 1a, 1b, 1c robot, 2, 2a, 2b control device, 3, 3a terminal device, 4 target, 11 drive mechanism, 12 sensor, 13, 34 camera, 14 robot arm, 15 hand, 16 dolly, 17 presentation device, 21 operation detection unit, 22 drive command generation unit, 23, 23a sensor image generation unit, 24 composite image generation unit, 25, 32 communication unit, 31 transparent contact panel, 33 display, 50 attachment, 100, 100a, 100b, 100c remote operation system.

Claims

1. Edge devices equipped with sensors, An operating device that receives the actions of an operator to operate the edge device as an operation to the edge device, A camera that captures camera footage which includes the surroundings of the edge device, A sensor image generation unit generates a sensor image, which is an image corresponding to the detected value, using the detected value detected by the sensor. An operation detection unit that detects the operation position, which is the position in the camera image corresponding to the operation, A composite image generation unit determines a position to superimpose the sensor image onto the camera image based on the aforementioned operation position, and generates a composite image by superimposing the sensor image onto the camera image at the determined position. A display device that displays the aforementioned composite image, Equipped with, The remote control system is characterized in that the composite image generation unit superimposes the sensor image onto the camera image such that the sensor image is superimposed on a position including the operation position.

2. The aforementioned operation is performed by the operator's finger. The operating device is a transparent contact panel integrated with the display device, The operation information received by the operating device includes position information indicating the position where the finger made contact with the transparent contact panel during the operation. The remote control system according to claim 1, characterized in that the operation detection unit sets the position in the camera image corresponding to the position indicated by the position information as the operation position, and uses the position information to determine the position in which the sensor image is superimposed on the camera image.

3. The remote control system according to claim 1 or 2, characterized in that the composite image generation unit superimposes the sensor image onto the camera image such that the center of the sensor image is the operation position.

4. The remote control system according to claim 1 or 2, characterized in that the composite image generation unit superimposes the sensor image onto the camera image such that the center of the sensor image is offset above the operation position.

5. The operation is performed by bringing the attachment, which is worn on the operator's finger, into contact with the transparent contact panel. The remote control system according to claim 2, characterized in that the composite image generation unit superimposes the sensor image onto the camera image such that the center of the sensor image is the operation position.

6. The operation described above is performed by a working part which is a part of the operator's body. The operating device includes a device for photographing the operator, The operation information received by the aforementioned operating device includes video footage captured by the aforementioned operating device. The remote control system according to claim 1, characterized in that the operation detection unit determines the operation position using the position of the operating part in the operation information received by the operation device.

7. The edge device is equipped with a hand capable of grasping a target, The sensor is attached to the hand, The remote control system according to claim 1 or 2, characterized in that the operation includes a gripping-related operation, which is an operation for gripping the target with the hand.

8. The remote control system according to claim 7, characterized in that the sensor includes a force-feed sensor.

9. The remote control system according to claim 7, characterized in that the sensor includes a temperature sensor.

10. The aforementioned operation is performed by the operator's finger. The operating device is a transparent contact panel integrated with the display device, The operation information received by the operating device includes position information indicating the position where the finger made contact with the transparent contact panel during the operation. The remote control system according to claim 7, characterized in that the operation detection unit determines that the operation performed by the operator is a gripping-related operation when the operator's two fingers touch the transparent contact panel simultaneously, and determines the position in which to superimpose the sensor image onto the camera image using the position information in the gripping-related operation.

11. The edge device is equipped with a moving mechanism for movement, The remote control system according to claim 1 or 2, characterized in that the sensor includes a sensor that detects the driving force that drives the moving mechanism.

12. A drive command generation unit that generates drive commands for the edge device, Equipped with, The remote control system according to claim 1 or 2, characterized in that the drive command generation unit generates a drive command corresponding to the operation information received by the operating device, thereby synchronizing the operation of the edge device with the operation of the operator.

13. The remote control system according to claim 1 or 2, characterized in that the composite image generation unit generates an image showing the distribution of the detected values ​​as the sensor image using the detected values ​​acquired by a plurality of identical sensors.

14. The remote control system according to claim 1 or 2, characterized in that the composite image generation unit generates sensor images that include images individually representing the detection values ​​of each type of sensor, using detection values ​​acquired by a plurality of different types of sensors.

15. The edge device is equipped with a collision sensor for detecting collisions. The remote control system according to claim 1 or 2, characterized in that the edge device stops operating when it detects a collision using the detection result of the collision sensor.

16. The remote control system according to claim 1 or 2, characterized in that the edge device includes a display device capable of displaying the detected value detected by the sensor.

17. A remote control method in a remote control system for remotely controlling edge devices equipped with sensors, A reception step that accepts the operator's actions to operate the edge device as an operation to the edge device, A shooting step of capturing camera footage which includes the surroundings of the edge device, A sensor image generation step, which generates a sensor image corresponding to the detected value using the detected value detected by the sensor, A detection step to detect the operation position, which is the position in the camera image corresponding to the operation, A composite image generation step involves determining the position where the sensor image is superimposed on the camera image based on the operation position, and generating a composite image by superimposing the sensor image on the camera image at the determined position. A display step of displaying the aforementioned composite image, Includes, The remote control method is characterized in that, in the composite image generation step, the sensor image is superimposed on the camera image such that the sensor image is superimposed on a position including the operation position.