Remote control system, remote control method and remote control program

Abstract

This invention provides a remote control system. The remote control system (100) includes an operating device (2), a robot (1), a contact force sensor (13), a camera device (71), a display device (8), and a control device (3). The operating device (2) is operated by a user. The robot (1) applies force to an object (W) according to the action of the operating device (2). The contact force sensor (13) is installed on the robot (1) to detect the action state of the robot (1). The camera device (71) captures images of at least one of the robot (1) and the object (W). The display device (8) displays the captured images by the camera device (71) and provides them to the user operating the operating device (2). The control device (3) executes motion control of at least one of the robot (1) and the operating device (2) according to the detection result of the contact force sensor (13). The control device (3) delays the motion control to reduce the deviation in the display timing of the captured images caused by the display device.

Description

Technical Field

[0001] This invention relates to a remote control system, a control method for a remote control system, and a remote control program. Background Technology

[0002] To date, the technology of remotely controlling a slave device via a master device is well known. For example, Patent Document 1 discloses a remote control system including a master device and a slave device, which are configured in physically separate locations. This remote control system includes a display device that displays images transmitted from the slave device to the master device. Furthermore, when there is a significant communication delay between the master and slave devices, the image is displayed blurry on the display device. Therefore, the user can be aware of situations with significant communication delays.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2015-47666 Summary of the Invention

[0004] As described above, in a remote control system, not only can control signals between the master and slave devices be sent and received, but also various signals such as image signals. The time required for signal transmission and reception, as well as the time required for signal processing, varies for each signal. If a time discrepancy occurs between each signal during the final signal processing, the user may experience discomfort. For example, in the case of the aforementioned remote control system, a time discrepancy between the control between the master and slave devices and the image display on the display device would likely cause user discomfort.

[0005] In view of the above, the purpose of the technology disclosed herein is to reduce the discomfort experienced by users in remote control systems that display captured images.

[0006] The remote control system disclosed herein includes a master device, a slave device, sensors, a camera device, a display device, and a control device. The master device is operated by a user. The slave device applies actions to an object based on the actions of the master device. The sensors are located on the slave device to detect its operational status. The camera device captures images of at least one of the slave device and the object. The display device displays the captured images and provides them to the user operating the master device. The control device executes action control of at least one of the master device and the slave device based on the detection results of the sensors. The control device delays the action control to reduce the timing deviation of the display of the captured images caused by the display device.

[0007] The remote control system method disclosed herein includes a master device, a slave device, a sensor, a camera device, and a display device. The master device is operated by a user. The slave device applies actions to an object based on the actions of the master device. The sensor is located on the slave device to detect its operational state. The camera device captures images of at least one of the slave device and the object. The display device displays the captured images and provides them to the user operating the master device. The remote control system method includes the steps of executing action control of at least one of the master device and the slave device based on the sensor detection results, and delaying the action control to reduce deviations in the display timing of the captured images caused by the display device.

[0008] The remote control program disclosed herein is a program for enabling a computer to control a remote control system. The remote control system includes a master device, slave devices, sensors, a camera device, and a display device. The master device is operated by a user. The slave devices apply actions to an object based on the actions of the master device. The sensors are located on the slave devices to detect their operational status. The camera device captures images of at least one of the slave devices and the object. The display device displays the captured images and provides them to the user operating the master device. The remote control program enables the computer to perform action control on at least one of the master device and slave devices based on the sensor detection results, and to delay the action control to reduce deviations in the timing of display of the captured images caused by the display device.

[0009] (Invention Effects)

[0010] According to the remote control system, the discomfort experienced by the user during the display of captured images can be reduced.

[0011] According to the control method of the remote control system, the discomfort experienced by the user in the remote control system that displays captured images can be reduced.

[0012] According to the remote control program, the discomfort experienced by the user in a remote control system that displays captured images can be reduced. Attached Figure Description

[0013] Figure 1 This is a schematic diagram illustrating the structure of the remote control system involved in the implementation method.

[0014] Figure 2 This is a diagram showing the general hardware structure of the robot control device.

[0015] Figure 3 This is a diagram showing the general hardware structure of the operation control device.

[0016] Figure 4 This is a diagram showing the general hardware structure of the control device.

[0017] Figure 5 This is a diagram showing the general hardware structure of an image processing device.

[0018] Figure 6 This is a diagram showing the general hardware structure of the display control device.

[0019] Figure 7 This is a block diagram showing the structure of the control system of the remote control system.

[0020] Figure 8 This is a flowchart illustrating the operation of the remote control system.

[0021] Figure 9 (i) is an example of a sensor signal output from the input processing unit. Figure 9 (ii) is an example of an image signal output from a camera device.

[0022] Figure 10 This is an example of an image signal output from the synthesis unit that synthesizes the sensor signal.

[0023] Figure 11 (i) is an example of a sensor signal obtained by the control device from the robot control device. Figure 11 (ii) is an example of an image signal decoded by a display control device.

[0024] Figure 12 This is an example of a sensor signal separated from an image signal. Detailed Implementation

[0025] Hereinafter, the exemplary embodiments will be described in detail with reference to the accompanying drawings. Figure 1 This is a schematic diagram illustrating the structure of the remote control system 100 according to the embodiment. It should be noted that... Figure 1 The dashed lines in the diagram represent wireless communication.

[0026] The remote control system 100 includes a master-slave system 110. The master-slave system 110 has an operating device 2, a robot 1, and a control device 3. The operating device 2 is operated by a user, the robot 1 applies actions to an object W according to the actions of the operating device 2, and the control device 3 controls both the robot 1 and the operating device 2. The operating device 2 and the control device 3 are configured at a first station S1. The robot 1 is configured at a second station S2, different from the first station S1. The master-slave system 110 enables remote control between the operating device 2 and the robot 1. The operating device 2 is an example of a master device, and the robot 1 is an example of a slave device.

[0027] It should be noted that, in this disclosure, the tasks performed by robot 1 do not include teaching tasks, confirmation of teaching, or correction of teaching tasks. Therefore, the operating device 2 does not include a teach pendant.

[0028] Robot 1 and operating device 2 are communicatively connected together. Specifically, robot 1 and control device 3 are wirelessly connected together. That is, signals are transmitted and received between robot 1 and control device 3 via a transmission path (hereinafter referred to as the "first transmission path") consisting of robot 1, wireless communication, and control device 3. Operating device 2 is connected to control device 3. That is, operating device 2 communicates with robot 1 via control device 3.

[0029] In this example, robot 1 is an industrial robot. Robot 1 applies a process to object W. Specifically, this process is machining, and more specifically, grinding. The process is not limited to grinding; it could also be milling or abrasion. Object W could be, for example, a large steel plate or the wall of a large tank.

[0030] Robot 1 has sensors for detecting the motion state of robot 1. In this example, the sensors also include a contact force sensor 13, which detects the reaction force (hereinafter referred to as "contact force") received from the object W.

[0031] The control device 3 receives the detection results from the contact force sensor 13 via the robot 1. Based on the detection results from the contact force sensor 13, the control device 3 executes motion control of at least one of the robot 1 and the operating device 2. In this example, the control device 3 controls the movement of the robot 1 and the movement of the operating device 2 based on the user's operation of the operating device 2 and the detection results from the contact force sensor 13, thereby indicating the reaction force acting on the robot 1 to the user.

[0032] The remote control system 100 also includes an image system 120. The image system 120 has a camera device 71 that captures images, and a display device 8 that displays the captured images. The image system 120 also has an image processing device 72 that processes the images captured by the camera device 71. The display device 8 is configured at a first station S1. The display device 8 is configured in a position visible to the user while operating the operating device 2. The camera device 71 and the image processing device 72 are configured at a second station S2. The camera device 71 captures images of the robot 1 and / or the object W.

[0033] The camera device 71 and the display device 8 are communicatively connected together. Specifically, the image processing device 72 and the display device 8 are wirelessly connected together. That is, signals are transmitted and received between the camera device 71 and the display device 8 via a transmission path (hereinafter referred to as the "second transmission path") through the image processing device 72, wireless communication, and the display device 8. The camera device 71 is connected to the image processing device 72. That is, the camera device 71 communicates with the display device 8 via the image processing device 72.

[0034] The image system 120 assists the user operating the operating device 2 by capturing images of the robot 1 and / or the object W using the camera device 71 and displaying the captured images using the display device 8.

[0035] (Detailed structure of the remote control system)

[0036] (robot)

[0037] Robot 1 may also have an end effector 11 and a robotic arm 12. The end effector 11 applies force to the object W, and the robotic arm 12 causes the end effector 11 to move. Robot 1 moves by moving the end effector 11 through the robotic arm 12, and applies force to the object W through the end effector 11. Robot 1 may also have a base 10 and a robot control device 14. The base 10 supports the robotic arm 12, and the robot control device 14 controls the entire robot 1.

[0038] A three-axis orthogonal robot coordinate system is defined for robot 1. For example, the Z-axis is set in the vertical direction, and the mutually orthogonal X-axis and Y-axis are set in the horizontal direction.

[0039] The end effector 11 has a grinding device 11a that applies grinding as an action to the object W. For example, the grinding device 11a can also be a grinding machine, a track sander, a random track sander, a triangular sander, or a belt sander, etc. The grinding machine can also be a type that rotates a disc-shaped grinding wheel, a type that rotates a conical or cylindrical grinding wheel, etc. Here, the grinding device 11a is a grinding machine.

[0040] The robotic arm 12 changes the position of the grinding device 11a. Furthermore, the robotic arm 12 can also change the posture of the grinding device 11a. The robotic arm 12 is a vertical multi-joint robotic arm. The robotic arm 12 has multiple links 12a, joints 12b, and servo motors 15 (see reference). Figure 2 The joint 12b connects to multiple links 12a, and the servo motor 15 rotates to drive the multiple joints 12b.

[0041] It should be noted that the robot arm 12 can also be a horizontal multi-joint robot arm, a parallel link robot arm, a Cartesian coordinate robot arm, or a polar coordinate robot arm, etc.

[0042] In this example, a contact force sensor 13 is positioned between the robot arm 12 and the end effector 11 (specifically, at the connection between the robot arm 12 and the end effector 11). The contact force sensor 13 detects the contact force received by the end effector 11 from the object W. The contact force sensor 13 detects forces in three orthogonal axial directions and torques about these three axes. The contact force sensor 13 is an example of a force sensor.

[0043] Furthermore, the force sensor is not limited to the contact force sensor 13. For example, the contact force sensor 13 may also detect forces in the direction of a single axis, dual axes, or three axes. Alternatively, the force sensor may be a current sensor that detects the current of the servo motor 15 of the robot arm 12, or a torque sensor that detects the torque of the servo motor 15, etc.

[0044] Figure 2 This diagram illustrates the schematic hardware structure of the robot control device 14. The robot control device 14 controls the servo motor 15 of the robot arm 12 and the grinding device 11a. The robot control device 14 receives detection signals from the contact force sensor 13. The robot control device 14 transmits and receives information, commands, and data from the control device 3 and the image processing device 72. The robot control device 14 communicates wirelessly with the control device 3. The robot control device 14 is connected to the image processing device 72 via wiring and transmits information and data to the image processing device 72 via the wiring. The robot control device 14 includes a control unit 16, a storage unit 17, a memory 18, and a communication unit 19.

[0045] The control unit 16 controls the entire robot control device 14. The control unit 16 performs various calculations. For example, the control unit 16 is formed using a processor such as a CPU (Central Processing Unit). The control unit 16 can also be formed using a MCU (Microcontroller Unit), MPU (Microprocessor Unit), FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), system LSI, etc.

[0046] The storage unit 17 stores programs and various data executed by the control unit 16. The storage unit 17 is formed using non-volatile memory, HDD (hard disk drive), or SSD (solid-state drive), etc.

[0047] Memory 18 temporarily stores data, etc. For example, memory 18 is formed using volatile memory.

[0048] Communication unit 19 is a communication module that performs wireless communication. For example, communication unit 19 implements wireless communication that conforms to 5G communication standards.

[0049] (Operating device)

[0050] like Figure 1 As shown, the operating device 2 includes an operating section 21 and an operating force sensor 23. The operating section 21 is operated by a user, and the operating force sensor 23 detects the operating force applied by the user to the operating section 21. The operating device 2 accepts input for manually operating the robot 1 and outputs operation information as the input information to the control device 3. Specifically, the user operates the operating device 2 by gripping the operating section 21. The operating force sensor 23 detects the force applied to the operating section 21 at this time. The operating force detected by the operating force sensor 23 is output as operation information to the control device 3.

[0051] The operating device 2 may also include a base 20, a support mechanism 22, and an operating control device 24. The support mechanism 22 is mounted on the base 20 and supports the operating part 21. The operating control device 24 controls the entire operating device 2. The operating device 2, under control from the control device 3, provides the user with feedback on the reaction force of the operating force. Specifically, the operating control device 24 receives commands from the control device 3 and controls the support mechanism 22 to allow the user to perceive the reaction force.

[0052] An orthogonal three-axis operating coordinate system is defined for the operating device 2. The operating coordinate system corresponds to the robot coordinate system. That is, the Z-axis is set in the vertical direction, and the mutually orthogonal X-axis and Y-axis are set in the horizontal direction.

[0053] The support mechanism 22 has multiple links 22a, joints 22b and servo motors 25 (see reference). Figure 3The joint 22b connects to multiple links 22a, and the servo motor 25 rotates to drive the multiple joints 22b. The support mechanism 22 supports the operating part 21, enabling the operating part 21 to be in any position and posture in three-dimensional space. The servo motor 25 rotates corresponding to the position and posture of the operating part 21. The amount of rotation of the servo motor 25, i.e., the rotation angle, is uniquely determined.

[0054] In this example, the operating force sensor 23 is disposed between the operating part 21 and the support mechanism 22 (specifically, at the connection between the operating part 21 and the support mechanism 22). The operating force sensor 23 detects the force in the directions of three orthogonal axes and the torque around these three axes.

[0055] Furthermore, the force detection unit is not limited to the force sensor 23. For example, the force sensor 23 may also detect forces in the direction of a single axis, dual axis, or three axes. Alternatively, the detection unit may be a current sensor that detects the current of the servo motor 25 of the support mechanism 22, or a torque sensor that detects the torque of the servo motor 25, etc.

[0056] Figure 3 This diagram illustrates the schematic hardware structure of the operation control device 24. The operation control device 24 controls the servo motor 25 to move the support mechanism 22. The operation control device 24 receives detection signals from the operating force sensor 23. The operation control device 24 is connected to the control device 3 via wiring, and transmits and receives information, commands, and data through the wiring. The operation control device 24 includes a control unit 26, a storage unit 27, and a memory 28.

[0057] The control unit 26 controls the entire operation control device 24. The control unit 26 performs various calculations. For example, the control unit 26 is formed using a processor such as a CPU (Central Processing Unit). The control unit 26 can also be formed using a MCU (Microcontroller Unit), MPU (Microprocessor Unit), FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), system LSI, etc.

[0058] The storage unit 27 stores programs and various data executed by the control unit 26. The storage unit 27 is formed using non-volatile memory, HDD (hard disk drive), or SSD (solid-state drive), etc.

[0059] Memory 28 temporarily stores data, etc. For example, memory 28 is formed using volatile memory.

[0060] (Control device)

[0061] Control device 3 controls the entire master-slave system 110, performing motion control on robot 1 and manipulator 2. Specifically, control device 3 performs master-slave control between robot 1 and manipulator 2, specifically, bidirectional control. Control device 3 executes a first control and a second control. The first control controls the movement of robot 1 based on the movement of manipulator 2 caused by user operation. The second control controls the movement of manipulator 2 to provide the user with feedback on the reaction force in response to the detection result of contact force sensor 13. In other words, through the first control, end effector 11 processes object W, and through the second control, the user is notified of the reaction force during processing.

[0062] Figure 4 This diagram illustrates the general hardware structure of the control device 3. The control device 3 communicates with the robot control device 14 and the operation control device 24, exchanging information, commands, and data. Furthermore, the control device 3 communicates with the display device 8, exchanging information, commands, and data. The control device 3 includes a control unit 31, a storage unit 32, a memory unit 33, and a communication unit 34. It should be noted that, although not shown, the control device 3 may also include an input operation unit and a display. The input operation unit is operated by the user to set the motion control settings for the robot 1 and the operation device 2, and the display shows the settings.

[0063] The control unit 31 controls the entire control device 3. The control unit 31 performs various calculations. For example, the control unit 31 is formed using a processor such as a CPU (Central Processing Unit). The control unit 31 can also be formed using a MCU (Microcontroller Unit), MPU (Microprocessor Unit), FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), system LSI, etc.

[0064] The storage unit 32 stores programs and various data executed by the control unit 31. For example, the storage unit 32 stores a remote control program for controlling the remote control system 100. The storage unit 32 is formed using non-volatile memory, HDD (hard disk drive), or SSD (solid-state drive), etc.

[0065] Memory 33 temporarily stores data, etc. For example, memory 33 is formed using volatile memory.

[0066] Communication unit 34 is a communication module that performs wireless communication. For example, communication unit 34 implements wireless communication that conforms to 5G communication standards.

[0067] (Camera device and image processing device)

[0068] The camera device 71 captures images of the robot 1 and the object W; specifically, it captures video. In this example, because the robot 1 is grinding the object W, the camera device 71 captures images centered on the end effector 11 and the object W. The camera device 71 captures video at a predetermined frame rate. The images captured by the camera device 71 are input as image signals to the image processing device 72.

[0069] The image processing device 72 receives the captured image (i.e., image signal) from the camera device 71, processes the image signal, and sends the processed image signal to the display device 8. The image processing device 72 encodes the processed image signal and sends the encoded image signal to the display device 8 via wireless communication.

[0070] In detail, the image processing device 72 receives the detection results from the contact force sensor 13 from the robot 1, in addition to the image signal from the camera device 71. As part of the image processing, the image processing device 72 associates the image captured by the camera device 71 and the detection results from the contact force sensor 13, acquired at corresponding moments. Specifically, the image processing device 72 appends the detection results from the contact force sensor 13, acquired at the moment corresponding to the captured image, to the captured image. The image processing device 72 encodes the captured image with the detection results from the contact force sensor 13 appended and sends it to the display device 8.

[0071] Figure 5 This diagram shows a schematic hardware structure of the image processing apparatus 72. The image processing apparatus 72 includes a control unit 73, a storage unit 74, a memory 75, and a communication unit 76.

[0072] The control unit 73 controls the entire image processing device 72. The control unit 73 performs various calculations. For example, the control unit 73 is formed using a processor such as a CPU (Central Processing Unit). The control unit 73 can also be formed using a MCU (Microcontroller Unit), MPU (Microprocessor Unit), FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), system LSI, etc.

[0073] The storage unit 74 stores programs and various data executed by the control unit 73. For example, the storage unit 74 is formed using non-volatile memory, HDD (hard disk drive), or SSD (solid-state drive).

[0074] The memory 75 temporarily stores data, etc. For example, the memory 75 is formed using volatile memory.

[0075] Communication unit 76 is a communication module that performs wireless communication. For example, communication unit 76 implements wireless communication that conforms to 5G communication standards.

[0076] (Display device)

[0077] The display device 8 displays images captured by the camera device 71. The display device 8 includes a display 81 and a display control device 82.

[0078] The display control device 82 receives images captured by the camera device 71 via wireless communication. Specifically, the display control device 82 communicates wirelessly with the image processing device 72. The display control device 82 receives image signals from the image processing device 72. The display control device 82 decodes the received image signals and outputs the decoded image signals to the display 81.

[0079] The display 81 displays the image signal input from the display control device 82 as an image.

[0080] Figure 6 This diagram illustrates the schematic hardware structure of the display control device 82. The display control device 82 includes a storage unit 83, a storage unit 84, a memory 85, and a communication unit 86.

[0081] The control unit 83 controls the entire image processing device 82. The control unit 83 performs various calculations. For example, the control unit 83 is formed using a processor such as a CPU (Central Processing Unit). The control unit 83 can also be formed using a MCU (Microcontroller Unit), MPU (Microprocessor Unit), FPGA (Field Programmable Gate Array), PLC (Programmable Logic Controller), system LSI, etc.

[0082] The storage unit 84 stores programs and various data executed by the control unit 83. For example, the storage unit 84 is formed using non-volatile memory, HDD (hard disk drive), or SSD (solid-state drive).

[0083] Memory 85 temporarily stores data, etc. For example, memory 85 is formed using volatile memory.

[0084] Communication Unit 86 is a communication module that performs wireless communication. For example, Communication Unit 86 implements wireless communication that conforms to 5G communication standards.

[0085] (Control of the remote control system)

[0086] In the remote control system 100 configured as described above, the control device 3 performs master-slave control, in which first control and second control are performed in parallel. In the first control, the robot 1's movement is controlled based on the action of the operating device 2 caused by the user's operation. In the second control, the operation of the operating device 2 is controlled to provide the user with a feedback force in response to the detection result of the contact force sensor 13. Simultaneously, the image system 120 performs image display control, in which the camera device 71 captures images of the robot 1 and the object W, and the captured images are displayed on the display device 8. The user can operate the operating device 2 while observing the captured images displayed on the display device 8. At this time, the control device 3 performs delay control to delay the master-slave control, thereby reducing the deviation between the master-slave control and the captured images displayed on the display device 8.

[0087] Furthermore, during the execution of master-slave control and image display control, the control device 3 calculates the time difference between the master-slave control and the image captured on the display device 8, and updates the delay amount of the delay control based on the time difference.

[0088] Figure 7 This is a block diagram showing the structure of the control system of the remote control system 100.

[0089] The control unit 16 of the robot control device 14 performs various functions by reading programs from the storage unit 17 into the memory 18 and expanding them. Specifically, the control unit 16 functions as the input processing unit 41 and the motion control unit 42.

[0090] The input processing unit 41 outputs information, data, and commands received from the contact force sensor 13 and the servo motor 15 to the control device 3. Specifically, the input processing unit 41 receives force detection signals (hereinafter referred to as "sensor signals") from the contact force sensor 13 along six axes and outputs these sensor signals to the control device 3. The sensor signals are transmitted to the control device 3 wirelessly. Furthermore, the input processing unit 41 receives detection signals from a rotation sensor (e.g., an encoder) and a current sensor from the servo motor 15. The input processing unit 41 outputs these detection signals to the motion control unit 42 for feedback control of the robot arm 12 performed by the motion control unit 42. The input processing unit 41 also outputs these detection signals as position information of the robot arm 12 to the control device 3. Additionally, the input processing unit 41 also outputs the sensor signals from the contact force sensor 13 to the image processing device 72.

[0091] The motion control unit 42 receives the command position xds from the control device 3 via wireless communication and generates control commands to move the robot arm 12 according to the command position xds. The motion control unit 42 outputs the control commands to the servo motor 15, causing the robot arm 12 to move and the grinding device 11a to move to the position corresponding to the command position. At this time, the motion control unit 42 provides feedback control of the robot arm 12's movement based on the detection signals from the rotation sensor and / or current sensor of the servo motor 15 from the input processing unit 41. Furthermore, the motion control unit 42 outputs the control commands to the grinding device 11a, causing the grinding device 11a to move. Therefore, the grinding device 11a grinds the object W.

[0092] The control unit 26 of the operation control device 24 performs various functions by reading programs from the storage unit 27 into the memory 28 and expanding them. Specifically, the control unit 26 functions as the input processing unit 51 and the motion control unit 52.

[0093] The input processing unit 51 outputs information, data, and commands received from the operating force sensor 23 to the control device 3. Specifically, the input processing unit 51 receives force detection signals from the operating force sensor 23 for all six axes and outputs these detection signals to the control device 3. Furthermore, the input processing unit 51 receives detection signals from the servo motor 25 from a rotation sensor (e.g., an encoder) and a current sensor. The input processing unit 51 outputs these detection signals to the motion control unit 52 for feedback control of the support mechanism 22 performed by the motion control unit 42.

[0094] The motion control unit 52 receives the command position xdm from the control device 3 and generates a control command to move the support mechanism 22 according to the command position xdm. The motion control unit 52 outputs the control command to the servo motor 25, causing the support mechanism 22 to move and the operating unit 21 to move to the position corresponding to the command position. At this time, the motion control unit 52 provides feedback control of the operation of the support mechanism 22 based on the detection signals from the rotation sensor and / or current sensor of the servo motor 25 from the input processing unit 51. Therefore, a reaction force is applied to the operating force applied by the user to the operating unit 21. As a result, the user can operate the operating unit 21 while simultaneously feeling a reaction force from the object W at the operating unit 21.

[0095] The control unit 31 of the control device 3 performs various functions by reading the program (specifically, the remote control program) from the storage unit 32 into the memory 33 and unfolding it. Specifically, the control unit 31 functions as an operating force acquisition unit 61, a contact force acquisition unit 62, an arithmetic unit 63, a force / velocity conversion unit 64, a first velocity / position conversion unit 65, a second velocity / position conversion unit 66, a separation unit 67, a deviation detection unit 68, and a delay unit 69.

[0096] Furthermore, the control unit 73 of the image processing apparatus 72 performs various functions by reading the program from the storage unit 74 into the memory 75 and expanding it. Specifically, the control unit 73 functions as the synthesis unit 77 and the encoder 78.

[0097] The control unit 83 of the display control device 82 performs various functions by reading programs from the storage unit 84 into the memory 85 and expanding them. Specifically, the control unit 83 functions as a decoder 87.

[0098] The operating force acquisition unit 61 receives the detection signal from the operating force sensor 23 via the input processing unit 51, and acquires the operating force fm based on the detection signal. The operating force acquisition unit 61 inputs the operating force fm into the calculation unit 63.

[0099] The contact force acquisition unit 62 receives the detection signal from the contact force sensor 13 via the input processing unit 41, and acquires the contact force fs based on the detection signal. The contact force acquisition unit 62 inputs the contact force fs to the calculation unit 63. In addition, the contact force acquisition unit 62 outputs the sensor signal to the deviation detection unit 68.

[0100] The calculation unit 63 calculates the sum of the operating force fm input from the operating force acquisition unit 61 and the contact force fs input from the contact force acquisition unit 62. Here, since the operating force fm and the contact force fs are forces in opposite directions, their signs are different. That is, by adding the operating force fm and the contact force fs, the absolute value of the operating force fm is reduced. The calculation unit 63 outputs the sum of the operating force fm and the contact force fs, i.e., the resultant force fm + fs.

[0101] The delay unit 69 has the function of delaying the master-slave control. In this example, the delay unit 69 is a low-pass filter, and more specifically, a first-order hysteresis filter. The first-order hysteresis filter contains a time constant. The delay unit 69 delays the resultant force fm+fs from the arithmetic unit 63 to the force / velocity conversion unit 64 according to the time constant.

[0102] The force / velocity conversion unit 64 converts the resultant force fm+fs input via the delay unit 69 into a command velocity xd'. The force / velocity conversion unit 64 calculates the command velocity xd' using a motion model based on a motion equation that includes the inertia coefficient, viscosity coefficient (damping coefficient), and stiffness coefficient (spring coefficient). Specifically, the force / velocity conversion unit 64 calculates the command velocity xd' according to the following motion equation.

[0103]

Mathematical Formula 1

[0104] md·xd″+cd·xd′+kd·xd=fm+fs…(1)

[0105] Here, xd is the command position. md is the inertia coefficient. cd is the viscosity coefficient. kd is the stiffness coefficient. fm is the operating force. fs is the contact force. Additionally, "'" indicates a single derivative, and "" indicates a double derivative.

[0106] Equation (1) is a linear differential equation. Solving equation (1) for xd' yields equation (2).

[0107]

Mathematical Formula 2

[0108] xd′=A…(2)

[0109] Here, A is a term represented by fm, fs, md, cd, kd, etc.

[0110] Equation (2) is stored in storage unit 32. Force / velocity conversion unit 64 reads equation (2) from storage unit 32, calculates command velocity xd', and outputs the calculated command velocity xd' to first velocity / position conversion unit 65 and second velocity / position conversion unit 66.

[0111] The first velocity / position conversion unit 65 uses the robot coordinate system as a reference to convert the command velocity xd' after coordinate transformation into the command position xds for robot 1. For example, when there is a ratio between the movement amount of robot 1 and the movement amount of operating device 2, the first velocity / position conversion unit 65 multiplies the command position xd obtained from the command velocity xd' according to the movement ratio to obtain the command position xds. The first velocity / position conversion unit 65 outputs the obtained command position xds to the robot control device 14, specifically, to the motion control unit 42. As described above, the motion control unit 42 causes the robot arm 12 to move according to the command position xds.

[0112] The second speed / position conversion unit 66 uses the operating coordinate system as a reference to convert the command speed xd' into the command position xdm used by the operating device 2. The second speed / position conversion unit 66 outputs the calculated command position xdm to the operating control device 24, specifically, to the motion control unit 52. As described above, the motion control unit 52 actuates the support mechanism 22 according to the command position xdm.

[0113] The compositing unit 77 adds the detection results of the contact force sensor 13, acquired at the time corresponding to the captured image, as correlation information indicating their mutual association to the captured image of the imaging device 71. Specifically, the compositing unit 77 receives an image signal from the imaging device 71 and a sensor signal from the contact force sensor 13 via the input processing unit 41. The compositing unit 77 combines the sensor signals with the image signal. The compositing unit 77 combines the sensor signals and image signals acquired by the contact force sensor 13 and the imaging device 71 at corresponding times (i.e., approximately the same times). For example, the compositing unit 77 considers the time when it receives the image signal and the sensor signal, respectively, as the respective acquisition times of the image signal and the sensor signal. In this way, the sensor signal acquired at the time corresponding to the image signal is added to the image signal as correlation information indicating their mutual association.

[0114] The sensor signal contains sensor signals of force along six axes. The synthesis unit 77 synthesizes at least one of the sensor signals of force along the six axes into the image signal. Preferably, the synthesis unit 77 synthesizes the sensor signal of force along the six axes that changes significantly when the robot 1 applies an action to the object W into the image signal. In this example, since the action applied to the object W is grinding, the synthesis unit 77 synthesizes the sensor signal of force along the Z-axis into the image signal.

[0115] At this time, the synthesis unit 77 uses electronic watermarking technology to synthesize the sensor signal into the image signal. Preferably, the synthesis unit 77 embeds the sensor signal into the image signal using a barely perceptible electronic watermark.

[0116] The encoder 78 encodes the image signal that has been synthesized from the sensor signal in the synthesis unit 77. The encoder 78 outputs the encoded image signal to the communication unit 76. The encoded image signal is then transmitted to the display control device 82 via the communication unit 76.

[0117] The display control device 82 receives image signals from the image processing device 72 via the communication unit 86. The image signals are input to the decoder 87. The decoder 87 decodes the image signals. The decoder 87 outputs the decoded image signals to the display 81 and the separation unit 67 of the control device 3.

[0118] The display 81 displays the image based on the decoded image signal.

[0119] The separation unit 67 separates the sensor signal from the decoded image signal. In this example, the sensor signal is synthesized into the image signal using electronic watermarking. The separation unit 67 separates the sensor signal from the image signal using electronic watermarking technology. The separation unit 67 outputs the separated sensor signal to the deviation detection unit 68.

[0120] The deviation detection unit 68 calculates and updates the delay time, which is the amount of delay that causes the master-slave control delay. The deviation detection unit 68 calculates the delay time based on the deviation time between the sensor signal received from the robot 1 and the image signal received from the display device 8. Here, the deviation time between the sensor signal and the image signal refers to the deviation in the time it takes for the sensor signal and image signal, which were originally acquired at approximately the same corresponding time, to be received by the deviation detection unit 68, i.e., the control device 3. Even if the sensor signal and image signal were originally acquired at approximately the same time, the time it takes to reach the control device 3 will deviate due to differences in the transmission path up to the control device 3 and the processing received along the way.

[0121] Furthermore, the sensor signals received from robot 1 are used to derive command position xds and command position xdm. Also, the image signal received from display device 8 is the same signal as the image signal output to display 81. In other words, the time difference between the sensor signal received from robot 1 and the image signal received from display device 8 corresponds to the time difference in display timing caused by display device 8 for master-slave control and image capture.

[0122] Specifically, the deviation detection unit 68 calculates the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal from the separation unit 67. The sensor signal is input to the deviation detection unit 68 simultaneously with the sensor signal from the contact force acquisition unit 62 and the sensor signal from the separation unit 67. The sensor signal from the contact force acquisition unit 62 is the sensor signal received from the robot control device 14, i.e., the robot 1. The sensor signal from the separation unit 67 is the signal separated from the decoded image signal in the display device 8, and is the signal acquired by the contact force sensor 13 at a time corresponding to the time when the image signal is acquired by the imaging device 71. Therefore, the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal from the separation unit 67 is equivalent to the deviation time between the sensor signal received from the robot 1 and the image signal received from the display device 8.

[0123] Furthermore, since the sensor signal synthesized in the image signal by the synthesis unit 77 is a sensor signal of force in the Z-axis direction that varies greatly during grinding among the 6-axis force sensor signals, the deviation detection unit 68 compares the sensor signal of force in the Z-axis direction from the sensor signal from the contact force acquisition unit 62 with the sensor signal from the separation unit 67.

[0124] The deviation detection unit 68 stores the obtained deviation time in the storage unit 32. The deviation time is accumulated in the storage unit 32.

[0125] The storage unit 32 stores a delay time that serves as a reference for determining the time constant of the delay unit 69. The initial value of the delay time is set by pre-calculating the deviation time as described above. The deviation detection unit 68 updates the delay time based on the deviation time accumulated in the storage unit 32. When the deviation time differs significantly from the delay time for a certain period of time, the deviation detection unit 68 updates the delay time with the deviation time.

[0126] When the delay time is updated, the deviation detection unit 68 calculates the time constant set in the delay unit 69 based on the delay time. Specifically, the storage unit 32 stores a conversion function for converting the delay time into a time constant. The deviation detection unit 68 uses the conversion function to calculate the time constant based on the delay time. The deviation detection unit 68 updates the time constant of the delay unit 69 with the calculated time constant. The delay unit 69 uses the updated time constant to perform the above-described process.

[0127] Next, the operation of the remote control system 100 configured in this way will be explained. Figure 8 This is a flowchart illustrating the operation of the remote control system 100.

[0128] (Actions of a master-slave system)

[0129] The master-slave control performed by the master-slave system 110 will be described. The master-slave control is achieved by the control device 3 reading the remote control program stored in the storage unit 32 into the memory 33 and unfolding it.

[0130] First, in step Sa0, delay control is set. The control device 3 displays a setting screen (illustration omitted) for setting the delay control. Specifically, the display shows whether delay control is enabled or disabled, and the delay amount (more specifically, delay time) for master-slave control when delay control is enabled. The user sets the delay control via the input operation unit.

[0131] Next, in step Sa1, when the user operates the operating device 2, the operating force sensor 23 detects the operating force applied by the user through the operating device 21. At this time, the contact force sensor 13 of the robot 1 detects the contact force.

[0132] The operating force detected by the operating force sensor 23 is input as a detection signal to the control device 3 via the input processing unit 51. In the control device 3, the operating force acquisition unit 61 inputs the operating force fm based on the detection signal to the calculation unit 63.

[0133] The contact force detected by the contact force sensor 13 is input as a sensor signal to the input processing unit 41. The sensor signal input to the input processing unit 41 is transmitted to the control device 3 via the communication unit 19. In addition, the input processing unit 41 inputs the sensor signal to the image processing device 72.

[0134] In the control device 3, the communication unit 34 receives sensor signals and inputs them to the contact force acquisition unit 62. The contact force acquisition unit 62 inputs the contact force fs based on the sensor signals to the calculation unit 63. Furthermore, the contact force acquisition unit 62 inputs the sensor signals to the deviation detection unit 68. The deviation detection unit 68 stores the sensor signals in the memory 33.

[0135] Secondly, in step Sa2, the control device 3 determines whether the delay control is effective.

[0136] When the delay control is invalid, in step Sa4, the grinding process performed by the robot 1 and the reaction force generated by the operating device 2 are prompted. Specifically, the calculation unit 63 calculates the resultant force fm+fs based on the input operating force fm and contact force fs, and inputs the resultant force fm+fs to the force / velocity conversion unit 64. At this time, the delay unit 69 is treated as an invalid component. That is, the resultant force fm+fs output from the calculation unit 63 is input to the force / velocity conversion unit 64 without delay. The force / velocity conversion unit 64 calculates the command velocity xd' using the resultant force fm+fs according to formula (2).

[0137] Regarding robot 1, the first speed / position conversion unit 65 calculates the command position xds based on the command speed xd'. The motion control unit 42 of robot control device 14 moves robot arm 12 according to the command position xds, controlling the position of grinding device 11a. Therefore, in response to the pressing force fm, a pressure is applied to the object W, and the object W is ground by grinding device 11a.

[0138] Regarding the operating device 2, the second speed / position conversion unit 66 calculates the command position xdm based on the command speed xd'. The action control unit 52 of the operating control device 24 actuates the support mechanism 22 according to the command position xdm, controlling the position of the operating unit 21. Therefore, the user perceives the reaction force in response to the contact force fs.

[0139] When the delay control is effective, the master-slave control is delayed in step Sa3. Specifically, the delay unit 69 delays the input of the combined force fm+fs from the arithmetic unit 63 to the force / velocity conversion unit 64 according to a set time constant. As a result, the time from the input of the detection signal from the operating force sensor 23 and the sensor signal from the contact force sensor 13 to the execution of motion control of the robot arm 12 and the operating unit 21 is delayed.

[0140] The master-slave system 110 repeats this process at a predetermined control cycle. The user can remotely control the robot 1, configured at the second station S2, to perform grinding of the object W by operating the operating device 2 at the first station S1. At that time, the user can operate the operating device 2 while sensing the reaction force received by the grinding device 11a from the object W. Furthermore, step Sa0 is executed only once at the start of the control and is not executed in subsequent control cycles.

[0141] (The action of the image system)

[0142] In parallel, the imaging system 120 captures images of the robot 1 and the object W, and performs image display control to display the captured images to the user.

[0143] First, in step Sb1, the camera device 71 captures images of the robot 1 and the object W. The camera device 71 captures video of the robot 1 and the object W at a predetermined frame rate. The camera device 71 then inputs the captured images (i.e., image signals) into the image processing device 72.

[0144] In step Sb2, the synthesis unit 77 of the image processing device 72 synthesizes the sensor signal from the contact force sensor 13 into the captured image. This synthesis is not for displaying the captured image to the user, but rather for updating the master-slave control delay time.

[0145] In detail, Figure 9 (i) is an example of a sensor signal output from the input processing unit 41. Figure 9 (ii) is an example of an image signal output from the camera device 71. Figure 10 This is an example of an image signal output from the synthesis unit 77 that has synthesized the sensor signal.

[0146] like Figure 9 As shown in (i), the sensor signal output from the input processing unit 41 includes data acquired during the sampling period corresponding to the control cycle of master-slave control. And as... Figure 9 As shown in (ii), the image signal input from the imaging device 71 contains frames (still images) acquired at a specified frame rate. In this example, the sampling period is shorter than the frame rate. Therefore, the compositing unit 77 combines the sensor signals acquired between each frame and the previous frame into the current frame. The compositing unit 77 combines the sensor signals as an image, i.e., a time-series signal waveform, into the images of each frame. The synthesis of the sensor signals is performed using electronic watermarking technology. As a result, as shown in (ii), the image signal is captured in the current frame. Figure 10As shown, the image that generates the sensor signal is used as an electronic watermark to synthesize the image signal in the frame. The synthesized frame and the sensor signal were acquired at approximately the same time.

[0147] In step Sb3, encoder 78 encodes the image signal synthesized by synthesis unit 77.

[0148] Then, in step Sb4, the image processing device 72 transmits the encoded image signal to the display control device 82 via wireless communication. The display control device 82 receives the image signal from the image processing device 72.

[0149] In the display control device 82, the decoder 87 decodes the image signal in step Sb5. The decoder 87 outputs the decoded image signal to the display 81.

[0150] In step Sb6, display 81 displays an image based on the image signal. Therefore, the captured image by camera device 71 is presented to the user. Here, display 81 displays the captured image synthesized with the sensor signal. However, since the sensor signal is synthesized using a subtle electronic watermarking technique, it is not visible to the user.

[0151] The image system 120 repeats this process at the frame rate of the captured images. The camera device 71 captures images of the robot 1 and the object W, which are positioned at the second station S2, and the display device 8 displays the captured images at the first station S1. The user can view the images of the robot 1 and the object W during or after grinding at the second station S2 using the display device 8 positioned at the first station S1, when or after operating the operating device 2.

[0152] (Delay control)

[0153] In this way, when master-slave control and image display control are performed in parallel, the time from when the camera device 71 acquires the captured image to when the display device 8 displays the captured image is longer than the time from when the sensor signal is acquired from the contact force sensor 13 to when the control device 3 uses the sensor signal to execute the motion control of the robot 1 and the operating device 2.

[0154] In detail, when robot 1 grinds object W, contact force sensor 13 detects the contact force from object W as a sensor signal, and camera device 71 acquires images of robot 1 and object W as image signals. The sensor signal from contact force sensor 13 is input to control device 3 via the first transmission path of master-slave system 110. As described above, control device 3 uses the sensor signal as one of the inputs to control robot 1 and operating device 2. The image signal from camera device 71 is input to display device 8 via the second transmission path of image system 120. Furthermore, the image signal is encoded in image processing device 72 and decoded in display control device 82 during this process.

[0155] Here, the data volume of the image signal is considerably larger compared to the sensor signal. This difference in data volume is especially pronounced with higher image quality. Therefore, the communication time for the image signal from the image processing device 72 to the display control device 82 is longer than the communication time for the sensor signal from the robot control device 14 to the control device 3. Furthermore, the image signal requires encoding and decoding. The larger the data volume of the image signal, the longer the encoding and decoding time.

[0156] As a result, regarding the sensor signals and captured images that were originally acquired at roughly the same time, for master-slave control using sensor signals, the timing deviation of the captured images caused by the display device 8, specifically, the delay. Figure 11 (i) is an example of a sensor signal obtained by the control device 3 from the robot control device 14. Figure 11 (ii) is an example of an image signal decoded by the display control device 82. For example... Figure 11 As shown in (i), due to factors such as the communication time between the robot control device 14 and the control device 3, the sensor signals acquired by the control device 3 and Figure 9 The sensor signal from the contact force sensor 13 output from the input processing unit 41, as shown in (i), is slightly delayed. However, this delay is so small as to be negligible. Figure 11 As shown in (ii), the decoded image signal is affected by factors such as encoding time, communication time between image processing device 72 and display control device 82, and decoding time, and... Figure 9 The image signal output from the camera device 71, as shown in (ii), is delayed. The delay of the image signal is greater than that of the sensor signal.

[0157] Therefore, grinding is performed by robot 1 and operating device 2 while the user observes images of robot 1 and object W that are delayed compared to real-time. Furthermore, the user receives two feedback messages from the image displayed on display device 8 and a feedback signal from the reaction force of operating device 2. However, the timing discrepancy between the feedback signal and the image display causes discomfort for the user. Moreover, people tend to rely more on visual information than tactile information. Therefore, the user operates operating device 2 based on the image displayed on display device 8, which is delayed compared to real-time, rather than the feedback signal from the reaction force. This tendency is greater with higher image quality. However, as mentioned above, higher image quality results in a greater delay in the image display from display device 8.

[0158] Therefore, in the remote control system 100, the control device 3 delays the master-slave control to reduce the deviation in the timing of image display caused by the display device 8 in master-slave control. As an example, in the master-slave control step Sa3, the delay unit 69 delays the input of the resultant force fm+fs from the arithmetic unit 63 to the force / velocity conversion unit 64. In this way, the control device 3 includes a delay member such as a low-pass filter of the delay unit 69 from the time the operating force and contact force are input until the robot 1 and the operating device 2 are moved. Therefore, the control device 3 delays the movement control of the robot 1 and the operating device 2, that is, it delays the time from the time the sensor signal from the contact force sensor 13 is received until the robot 1 and the operating device 2 are moved.

[0159] This means that the responsiveness of master-slave control decreases. However, the responsiveness of image display control (i.e., the responsiveness of image display performed by display device 8 to image capture performed by camera device 71) is consistent with or close to that of master-slave control. Even when the user relies on the captured image displayed on display device 8 to operate operation device 2, discomfort to the user can be reduced. For example, it is possible to prevent the object W from being ground beyond the range displayed on display device 8.

[0160] Especially when capturing high-quality images, the time required for image signal encoding, communication, and decoding increases. Even with advancements in communication technology that have shortened communication times, the time required for image signal encoding and decoding remains significant. Furthermore, the high quality of captured images increases the user's reliance on the captured images when operating the control device 2. Consequently, the user is more likely to perceive deviations in the timing of image display caused by the display device 8 under master-slave control. The remote control system 100 reduces these deviations in timing by delaying master-slave control, allowing for a decrease in responsiveness, while simultaneously reducing the discomfort experienced when operating the control device 2 while viewing the captured images on the display device 8.

[0161] (Exporting Delay Quantity)

[0162] Furthermore, during such master-slave control and image display control, the control device 3 calculates the delay amount of master-slave control in the delay control.

[0163] In detail, according to Figure 6 To illustrate with a flowchart, in step Sa1, the input processing unit 41 sends the sensor signal of the contact force sensor 13 to the control device 3 via the communication unit 19, and inputs it to the image processing device 72. In the image processing device 72, as described above, in step Sb2, the synthesis unit 77 associates the captured images from the camera device 71 acquired at corresponding times with the detection results of the contact force sensor 13. Specifically, the image processing device 72 appends the sensor signal of the contact force sensor 13 acquired at the time corresponding to the captured image to the captured image. More specifically, the synthesis unit 77 uses electronic watermarking technology to synthesize the sensor signals acquired at approximately the same time as each frame into the images of each frame of the image signal.

[0164] In the image system 120, the image signal synthesized with the sensor signal is encoded (step Sb3), wirelessly transmitted (step Sb4), and decoded (step Sb5). Then, the decoded image signal is input not only from the display control device 82 to the display 81, but also from the display control device 82 to the control device 3.

[0165] In the master-slave system 110, the separation unit 67 is as follows: Figure 11 In the decoded image signal as shown in (ii), Figure 12The sensor signal is separated as shown. In this example, the sensor signal is separated from the image signal using electronic watermarking technology. The separated sensor signal is input to the deviation detection unit 68 from the separation unit 67. The sensor signal input from the input processing unit 41 to the control device 3 is also input to the deviation detection unit 68 via the contact force acquisition unit 62. That is, the sensor signal of the contact force sensor 13 is input to the deviation detection unit 68 via two transmission paths: a first transmission path, a second transmission path, and a third transmission path. The first transmission path is the robot 1, wireless communication, and control device 3, and the second transmission path is the image processing device 72, wireless communication, and display control device 82. As described above, the sensor signal via the second transmission path undergoes encoding and decoding processing along this route.

[0166] Furthermore, in step Sa5, the deviation detection unit 68 determines whether the prescribed detection conditions are met. In this example, the detection condition is that the deviation detection unit 68 receives a sensor signal from the separation unit 67.

[0167] Because the control cycle of master-slave control is shorter than the cycle corresponding to the frame rate of the image signal, the deviation detection unit 68 receives the sensor signal from the contact force acquisition unit 62 (i.e., the sensor signal via the first transmission path) with a shorter cycle than the sensor signal from the separation unit 67 (i.e., the sensor signal via the second transmission path). Therefore, when the detection condition is not met, that is, when no sensor signal is input from the separation unit 67 to the deviation detection unit 68, the deviation detection unit 68 stores the sensor signal input from the contact force acquisition unit 62 in the memory 33, ending the derivation of the delay amount for this control cycle.

[0168] When the detection conditions are met, that is, when the sensor signal is input from the separation unit 67 to the deviation detection unit 68, the deviation detection unit 68 compares the sensor signal from the separation unit 67 with the sensor signal from the contact force acquisition unit 62 stored in the memory 33 in step Sa6, and calculates the deviation time of the sensor signal from the separation unit 67 relative to the sensor signal from the contact force acquisition unit 62.

[0169] Specifically, such as Figure 12As shown, the deviation detection unit 68 periodically, specifically at the frame rate of the captured image, inputs sensor signals from the separation unit 67. The sensor signals, as shown in 11(i), are stored in the memory 33. For example, the deviation detection unit 68 searches for a signal waveform from the sensor signals from the contact force acquisition unit 62 stored in the memory 33 that matches the signal waveform of the sensor signal from the separation unit 67. Furthermore, the deviation detection unit 68 calculates the deviation time as the time difference between the portions of the signal waveforms matching those of the sensor signals from the separation unit 67 and the contact force acquisition unit 62. The deviation detection unit 68 stores the calculated deviation time in the memory 32.

[0170] The deviation detection unit 68 repeats this deviation time derivation every time the detection conditions are met. As a result, the deviation time is accumulated in the storage unit 32.

[0171] In step Sa7, the deviation detection unit 68 determines whether the specified update conditions are met. For example, the update condition is that the state in which the deviation time changes by more than a specified range (hereinafter referred to as the "variation threshold") with respect to the current delay time continues for a specified period of time (hereinafter referred to as the "period threshold"). In short, the update condition is that the state in which the deviation time is significantly different from the current delay time continues for a certain period of time.

[0172] In detail, when the deviation detection unit 68 calculates the deviation time, it compares the deviation time with the current delay time to determine whether the time difference exceeds a variation threshold, and stores the determination result in the storage unit 32. Furthermore, when the time difference exceeds the variation threshold, the deviation detection unit 68 reads the determination result from the storage unit 32 for the period tracing back from the current period threshold, and determines whether the state of the time difference exceeding the variation threshold has continued beyond the period threshold. It should be noted that positive or negative changes in the time difference during the period threshold are not considered as a continuation of the state of the time difference exceeding the variation threshold.

[0173] When the time difference exceeds the change threshold and the state does not continue above the threshold for a period of time, the deviation detection unit 68 ends the derivation of the delay amount in this control cycle.

[0174] When the time difference exceeds the variation threshold and persists for more than a certain period, the deviation detection unit 68 averages the deviation time during that period threshold in step Sa8 and updates the current delay time with this average value. In this example, the delay time corresponds to the delay amount in master-slave control in delay control.

[0175] The deviation detection unit 68 uses the conversion function stored in the storage unit 32 to calculate the time constant based on the updated delay time. The deviation detection unit 68 then updates the time constant of the delay unit 69 with the calculated time constant.

[0176] At this time, the deviation detection unit 68 sets a limit on the range of change of the delay time. That is, in order to prevent the responsiveness of the master-slave control from changing drastically, the deviation detection unit 68 gradually changes the current delay time into a new delay time over multiple control cycles.

[0177] In this way, during master-slave control and image display control, the remote control system 100 calculates the delay amount of master-slave control using the signals of master-slave control and image display control. Specifically, the deviation detection unit 68 of the control device 3 calculates the deviation time between the sensor signal and the image signal, and calculates the delay amount based on the calculated deviation time. The sensor signal is the signal received by the control device 3 from the contact force sensor 13 for master-slave control, and the image signal is the signal transmitted from the camera device 71 to the display device 8 for image display control. Since the delay amount is calculated based on the signals that are transmitted and processed for master-slave control and image display control during the execution of master-slave control and image display control, a delay amount that conforms to the actual control situation can be calculated. For example, the communication environment between the robot control device 14 and the control device 3, and the communication environment between the image processing device 72 and the display control device 82, may change. Even when these communication environments change, the delay amount can be calculated using the above method to change the delay amount accordingly to the change in the communication environment.

[0178] Furthermore, the encoding and decoding times may vary depending on the image captured by the camera device 71. For example, the encoding and decoding times differ between images with uniform color and those with different colors and brightness. During grinding, sparks may be generated at the contact point between the object W and the grinding device 11a. In other words, the encoding and decoding times differ depending on the presence or amount of sparks in the image. Even in this case, the delay amount can be calculated using the method described above, and the delay amount can be adjusted accordingly to the image condition.

[0179] Furthermore, by pre-associating the sensor signals and image signals acquired by the contact force sensor 13 and the camera device 71 at corresponding times, the control device 3 can identify the associated sensor signals and image signals from the sensor signals used in master-slave control and the image signals displayed on the display device 8, and calculate those deviation times.

[0180] Specifically, as correlation information indicating their interrelationship, the sensor signal is attached to the image signal in the sensor signal and image signal acquired by the contact force sensor 13 and the imaging device 71 at corresponding moments. The control device 3 calculates the deviation time between the sensor signal and the image signal based on the correlation information. That is, when the control device 3 receives both the sensor signal and the image signal, it can determine the image signal acquired at the moment corresponding to the sensor signal by comparing the sensor signal and the sensor signal attached to the image signal, and can calculate the deviation time between the two. In this example, the control device 3 separates the sensor signal from the image signal by the separation unit 67, and calculates the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal separated from the image signal as the deviation time between the sensor signal and the image signal.

[0181] At this point, by using the sensor signal as the correlation information, there is no need to attach other information to the sensor signal; only the sensor signal needs to be attached to the image signal. Furthermore, no other information besides the sensor signal and image signal needs to be prepared. Therefore, the processing is simplified.

[0182] Furthermore, when multiple remote control systems 100 are configured, contamination may occur in wireless communication. Even in this case, by using sensor signals as correlation information, it is easy to determine whether the combination of sensor signals and image signals is appropriate (i.e., whether contamination has occurred). For example, when the acquisition time is used as correlation information and the acquisition time information is appended to each of the sensor signal and image signal, identification information for identifying each of the remote control systems 100 needs to be further appended to the sensor signal and image signal. When the sensor signal is used as correlation information, the sensor signal functions as identification information for the remote control system 100. Therefore, additional identification information for identifying each of the remote control systems 100 is not required.

[0183] Furthermore, in the remote control system 100, the addition of sensor signals to the image signal is performed before the image signal is encoded, and the separation of sensor signals from the image signal is performed after the image signal is decoded. Therefore, the sensor signals separated by the separation unit 67 delay the time required for image signal encoding, wireless communication of the image signal, and image signal decoding. Thus, the control device 3 can determine the deviation time that appropriately reflects the effects of image signal encoding, wireless communication, and decoding.

[0184] Furthermore, the addition of sensor signals to the image signal is achieved through electronic watermarking technology. Therefore, even if the display device 8 displays the image signal synthesized with the sensor signal on the monitor 81, the sensor signal will not become an obstacle to prompting the user about the captured image. In other words, in the display device 8, there is no need for special processing such as separating the sensor signal from the image signal.

[0185] Furthermore, when the deviation time exceeds a predetermined threshold for a specified period of time, the control device 3 updates the master-slave control delay amount, i.e., the delay time. Therefore, it can ignore the small fluctuations in deviation time allowed in normal master-slave control and image display control, keeping the master-slave control delay amount constant and thus stabilizing the master-slave control. In other words, the communication environment between the image processing device 72 and the display control device 82 is not constant and may vary to some extent. Moreover, the time required for encoding and decoding image signals varies depending on the captured image. Therefore, when the fluctuation in deviation time is small or temporary, the control device 3 can maintain a constant master-slave control delay amount, thereby maintaining constant responsiveness and preventing user discomfort.

[0186] Furthermore, when updating the delay time, the control device 3 can reduce user discomfort by gradually changing the delay time. In other words, the delay time is directly related to the responsiveness of the master-slave control. The control device 3 can prevent drastic changes in the responsiveness of the master-slave control by gradually changing the delay time.

[0187] As described above, the remote control system 100 includes an operating device 2 (master device), a robot 1 (slave device), a contact force sensor 13 (sensor), a camera device 71, a display device 8, and a control device 3. The operating device 2 is operated by a user. The robot 1 applies force to the object W according to the operation of the operating device 2. The contact force sensor 13 is installed on the robot 1 and detects the robot 1's movement state. The camera device 71 captures images of at least one of the robot 1 and the object W. The display device 8 displays the captured images and provides prompts to the user operating the operating device 2. The control device 3 executes motion control of at least one of the robot 1 and the operating device 2 based on the detection results of the contact force sensor 13. The control device 3 delays the motion control to reduce the timing deviation of the displayed images caused by the display device 8.

[0188] In other words, the control method of the remote control system 100, which includes an operating device 2, a robot 1, a contact force sensor 13, a camera device 71, and a display device 8, includes the steps of performing motion control on at least one of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13, and delaying the motion control to reduce the deviation in display timing caused by the display device 8 for the motion control and the captured image, wherein the operating device 2 is operated by a user, the robot 1 applies action to the object W according to the action of the operating device 2, the contact force sensor 13 is provided on the robot 1 and detects the motion state of the robot 1, the camera device 71 captures images of at least one of the robot 1 and the object W, and the display device 8 displays the captured images captured by the camera device 71 and provides them to the user operating the operating device 2.

[0189] Alternatively, a remote control program for enabling a computer to control the remote control system 100 enables the computer to perform motion control of at least one of the robot 1 and the operating device 2 based on the detection results of the contact force sensor 13, and to delay the motion control to reduce the display timing deviation caused by the display device 8 for the motion control and the captured images. The remote control system 100 includes the operating device 2, the robot 1, the contact force sensor 13, the camera device 71, and the display device 8. The operating device 2 is operated by a user. The robot 1 applies action to the object W according to the action of the operating device 2. The contact force sensor 13 is installed on the robot 1 and detects the motion state of the robot 1. The camera device 71 captures images of at least one of the robot 1 and the object W. The display device 8 displays the captured images by the camera device 71 and provides them to the user operating the operating device 2.

[0190] According to these structures, the robot 1 is operated by the user to apply an action (e.g., grinding) to the object W by manipulating the operating device 2. At this time, the contact force sensor 13 detects the motion state of the robot 1 (e.g., the reaction force acting on the robot 1). Motion control of at least one of the robot 1 and the operating device 2 is executed based on the detection result of the contact force sensor 13. In parallel, the camera device 71 captures an image of at least one of the robot 1 and the object W, and the display device 8 displays the captured image, providing it to the user. Here, the amount of data in the captured image is relatively large compared to the detection result of the contact force sensor 13. Therefore, the processing and communication time required from the time the captured image is acquired by the camera device 71 to its display on the display device 8 is longer than the processing and communication time required from the time the detection result is acquired by the contact force sensor 13 to its use in motion control. In other words, for motion control of at least one of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13, the timing of the display of the captured image by the display device 8 tends to be biased (specifically, delayed). To address this, motion control is delayed to reduce the timing deviation of the display of the captured image caused by the display device 8. Therefore, although the responsiveness of motion control decreases, it reduces discomfort for the user operating the operation device 2 while viewing the captured image on the display device 8.

[0191] Furthermore, the control device 3 receives the detection results of the contact force sensor 13 from the robot 1 and receives the captured image from the display device 8, and calculates the delay amount that causes the motion control to be delayed based on the deviation time between the detection results of the contact force sensor 13 and the captured image.

[0192] According to this structure, the control device 3 receives the detection result of the contact force sensor 13 from the robot 1 and performs motion control based on the detection result. On the other hand, the control device 3 receives a captured image from the display device 8, which is the image displayed on the display device 8. That is, the time difference between the detection result of the contact force sensor 13 received by the control device 3 and the captured image is approximately the same as the time difference between the motion control and the display of the captured image by the display device 8. Therefore, the control device 3 can calculate the motion control delay corresponding to the deviation between the motion control and the display of the captured image by the display device 8 by calculating the motion control delay based on the time difference between the detection result of the contact force sensor 13 and the captured image.

[0193] Furthermore, the control device 3 determines the deviation time by adding correlation information to at least one of the detection results and captured images obtained by the contact force sensor 13 and the camera device 71 at corresponding times.

[0194] According to this structure, the control device 3 can easily distinguish the detection results and captured images obtained at corresponding times from the detection results and captured images by referring to the associated information.

[0195] Furthermore, the remote control system 100 also includes an image processing device 72. The image processing device 72 receives captured images from the camera device 71, processes the captured images, and sends the processed captured images to the display device 8. The image processing device 72 receives detection results from the robot 1, appends the detection results obtained at the time corresponding to the captured image as associated information to the captured image, and sends the captured image with the appended detection results to the display device 8. The control device 3 receives the captured image with the appended detection results from the display device 8 and calculates the deviation time based on the comparison between the detection results appended to the captured image and the detection results received from the robot 1.

[0196] According to this structure, the detection result of the contact force sensor 13, which serves as correlation information representing the relationship between the two, is appended to the detection result of the contact force sensor 13 and the captured image of the imaging device 71 acquired at corresponding moments. Therefore, the detection result is appended to the captured image sent to the display device 8 and received by the control device 3 from the display device 8. The control device 3 receives the detection result from the robot 1. The deviation time between the detection result received from the robot 1 and the captured image with the detection result appended is the deviation time between the detection result and the captured image. Therefore, the control device 3 can easily determine the deviation time between the detection result and the captured image by comparing the detection result appended to the captured image and the detection result received from the robot 1.

[0197] Furthermore, the image processing device 72 appends the detection results to the captured image by synthesizing the detection results acquired at the time corresponding to the captured image as an image, and then encodes the captured image with the detection results and sends it to the display device 8. The control device 3 obtains the detection results from the captured image decoded by the display device 8.

[0198] According to this structure, the detection result of the contact force sensor 13, which is received by the control device 3 from the display device 8 and synthesized into the captured image, undergoes encoding, communication between the image processing device 72 and the display device 8, and decoding together with the captured image. In other words, the control device 3 can calculate the deviation time, which reflects the delay caused by these processes and communications, by comparing the detection result attached to the decoded captured image with the detection result received from the robot 1.

[0199] The control device 3 acts as a motion controller, controlling the actions of the robot 1 and the actions of the operating device 2 based on the user's operation of the operating device 2 and the detection results, so as to inform the user of the reaction force acting on the robot 1.

[0200] According to this structure, the control device 3 includes the control of the robot 1's movements and the control of the operating device 2's movements in motion control. That is, when a deviation occurs in the display timing of the captured image caused by the display device 8 for motion control, there is a deviation between the control of the robot 1's movements and the display timing of the captured image displayed by the display device 8, and also a deviation between the control of the operating device 2's movements and the display timing of the captured image displayed by the display device 8. For example, when the display timing of the captured image displayed by the display device 8 is delayed for the control of the robot 1's movements, the user operates the operating device 2 while observing the robot 1's movements or the captured image of the object W, which is delayed compared to the actual movements of the robot 1. Furthermore, when the display timing of the captured image displayed by the display device 8 is delayed for the control of the operating device 2's movements, the user perceives the reaction force from the operating device 2 while observing the robot 1's movements or the captured image of the object W, which is delayed compared to the time when the user receives the reaction force prompted by the operating device 2. In either case, the user will feel uncomfortable. Therefore, since the control device 3 delays the motion control, both the control of the robot 1's movements and the control of the operating device 2's movements are delayed. As a result, when the user operates the operating device 2 while observing the captured image on the display device 8, the deviation between the robot 1's movements and the captured image, as well as the reaction force felt from the operating device 2 and the deviation between the captured image, can be reduced.

[0201] The contact force sensor 13 is an example of a sensor that detects the motion state of robot 1, and also an example of a force sensor.

[0202] (Other implementation methods)

[0203] As described above, the embodiments have been presented as examples of the technology disclosed in this application. However, the technology disclosed herein is not limited to this and can be applied to embodiments with appropriate modifications, substitutions, additions, omissions, etc. Furthermore, the various constituent elements described in the embodiments can be combined to form new embodiments. Moreover, the constituent elements described in the drawings and detailed descriptions include not only those necessary to solve the problem, but also, for the purpose of illustrating the technology, constituent elements that are not necessary to solve the problem. Therefore, one should not immediately assume that those non-essential constituent elements are essential simply because they are described in the drawings and detailed descriptions.

[0204] The structure of the remote control system 100 described herein is merely an example and is not limited thereto. For instance, regarding the master-slave system 110, the action applied by the robot 1 to the object W is not limited to grinding. The action applied by the robot 1 to the object W can be cutting or grinding, or pressing or holding, etc.

[0205] Furthermore, the communication between robot 1 and control device 3 is not limited to wireless communication, but can also be wired communication. Similarly, the communication between camera device 71 and display device 8 is not limited to wireless communication, but can also be wired communication.

[0206] The motion control of robot 1 and manipulator 2 performed by control device 3 is not limited to bidirectional control. For example, control device 3 can also independently perform motion control of robot 1 in response to the action of manipulator 2 and motion control of manipulator 2 in response to the reaction force received by robot 1 from object W.

[0207] Alternatively, the control device 3 may only control the robot 1 in response to the action of the operating device 2, without providing feedback on the reaction force to the user via the operating device 2. In other words, the control device 3 may only control the robot 1 based on the user's operation of the operating device 2 and the sensor signal from the contact force sensor 13.

[0208] Furthermore, Robot 1 is not limited to industrial robots. For example, Robot 1 could also be a medical robot. Also, the subordinate device is not limited to robots. For example, the subordinate device could also be machinery such as construction machinery.

[0209] The sensor used to detect the motion state of the slave device is not limited to the contact force sensor 13. Any sensor that can detect the motion state of the slave device can be used. For example, when the slave device is robot 1, the sensor used to detect the motion state of the slave device can also be a current sensor that detects the current of the servo motor 15 or a torque sensor that detects the torque of the servo motor 15, etc.

[0210] Furthermore, the sensor signals synthesized in the captured image are not limited to force sensor signals in the Z-axis direction. The sensor signals synthesized in the captured image can be force sensor signals in directions other than the Z-axis direction, or they can include force sensor signals in multiple directions not limited to the Z-axis direction. For example, the user can select the sensor signals synthesized in the captured image from the 6-axis force sensor signals in step Sa0, etc. Alternatively, the robot control device 14 or the image processing device 72 can also select the sensor signals synthesized in the captured image from the 6-axis force sensor signals. For example, the robot control device 14 or the image processing device 72 can also select the sensor signal with larger variations from the 6-axis force sensor signals of the contact force sensor 13.

[0211] The synthesis of sensor signals in captured images is not limited to imperceptible electronic watermarking techniques. For example, sensor signals can also be synthesized in a visible state in captured images. Synthesis of sensor signals in a visible state can be achieved using both perceptible electronic watermarking techniques and other methods.

[0212] In the above example, the sensor signal attached to each frame of the image signal contains multiple numerical data, forming a signal waveform that can be distinguished from the sensor signals attached to other frames. However, depending on the sampling period of the sensor signal and the frame rate of the image signal, the sensor signal attached to each frame of the image signal may have only a few numerical data, and sometimes may not form a signal waveform that can be distinguished from the sensor signals attached to other frames. In this case, the deviation detection unit 68 can also combine the sensor signals attached to one or more immediately preceding frames with the sensor signals attached to the frame from which the deviation time is derived, and compare the combined sensor signal with the sensor signal from the contact force acquisition unit 62 while a distinguishable signal waveform has been formed.

[0213] The correlation information indicating the relationship between the sensor detection results acquired at corresponding moments and the image captured by the camera device is not limited to the sensor detection results (in the example above, the sensor signal of the contact force sensor 13). For example, the correlation information could also be time information. That is, the time information of the acquisition of the detection result could be appended to the sensor detection result, and the time information of the acquisition of the image could be appended to the image captured by the camera device. In that case, it is not necessary to append the sensor detection result to the image signal. For example, the robot control device 14 could append the time information of the acquired sensor signal to the sensor signal of the contact force sensor 13, and the image processing device 72 could append the time information of the acquired image signal to the image signal of the camera device 71. It is not necessary to input the sensor signal from the robot control device 14 to the image processing device 72. The deviation detection unit 68 can determine the deviation time between the sensor signal and the image signal acquired at corresponding moments by comparing the time signal appended to the sensor signal and the time signal appended to the image signal. Adding time information to an image signal can be done either by adding the time information as data or by incorporating the time information into the captured image. Additionally, associated information can be identification information beyond the time information, indicating the timing of the acquisition.

[0214] The update condition for the delay time is not limited to the state where the deviation time changes by more than a specified range relative to the current delay time for a specified period of time. For example, the update condition could also be that the deviation time changes by more than a specified range relative to the current delay time. Alternatively, the update condition could be unrelated to the change in deviation time, but rather the arrival of a specified period. That is to say, the delay time can also be updated at each specified period.

[0215] When updating the delay time based on the condition that the deviation time exceeds a specified range for a specified period of time, the new delay time may not be the average of the deviation times over the specified period. For example, the new delay time may be the latest deviation time or the deviation time with the highest frequency during the specified period.

[0216] Furthermore, when updating the delay time, it is not necessary to gradually update the current delay time to the new delay time, but to update it all at once.

[0217] Furthermore, the update of the delay time may not be performed automatically by the control device 3, but rather the user will be prompted with a new delay time when the update conditions are met, and the delay time will be updated through user operation.

[0218] The above block diagram is an example. It can also be implemented by combining multiple blocks into one block, dividing one block into multiple blocks, or transferring some functions to other blocks.

[0219] The technology disclosed herein can be either a program for executing the control method or a non-transitory computer-readable recording medium on which the program is recorded. Furthermore, the program can also be a program distributed via a transmission medium such as a network.

[0220] The functions of the structures disclosed in this embodiment can also be executed using circuits or processing circuits. A processor is a processing circuit, etc., that includes transistors and other circuits. In this disclosure, the unit, controller, or device is hardware or programmed to perform the described functions. Here, hardware refers to the hardware disclosed in this embodiment or known hardware that is configured or programmed to perform the functions disclosed in this embodiment. When the hardware is a processor or controller, the circuit, device, or unit is a combination of hardware and software, with software used to construct the hardware and / or processor.

Claims

1. A remote control system, characterized in that: The remote control system includes a master device, slave devices, sensors, a camera device, a display device, and a control device. The master device is operated by a user. The slave devices apply actions to an object based on the actions of the master device. The sensors are installed on the slave devices to detect their operational status. The camera device captures images of at least one of the slave devices and the object. The display device displays the captured images and provides them to the user operating the master device. The control device executes action control of at least one of the master device and slave devices based on the sensor detection results. The control device delays the motion control to reduce the timing deviation of the display of the captured image caused by the display device in relation to the motion control. The control device receives the detection result from the slave device and the captured image from the display device. By making the signal waveform of the captured image consistent with the signal waveform of the corresponding detection result, the delay amount for delaying the action control is calculated based on the time deviation between the detection result and the captured image.

2. The remote control system according to claim 1, characterized in that: When the control device updates the delay amount with a new delay amount calculated based on the deviation time, it gradually updates the delay amount to the new delay amount.

3. The remote control system according to claim 1, characterized in that: At least one of the detection results acquired by the sensor and the camera device at corresponding moments and the captured image is supplemented with correlation information indicating their mutual relationship. The control device calculates the deviation time by identifying the detection results and the captured images obtained at corresponding times from the received detection results and the captured images based on the association information.

4. The remote control system according to claim 3, characterized in that: The remote control system further includes an image processing device, which receives the captured image from the camera device, processes the captured image, and sends the processed captured image to the display device. The image processing device receives the detection result from the slave device, appends the detection result acquired at a time corresponding to the captured image as the association information to the captured image, and sends the captured image with the appended detection result to the display device. The control device receives the captured image with the detection result attached from the display device, and calculates the deviation time by comparing the detection result attached to the captured image with the detection result received from the slave device.

5. The remote control system according to claim 4, characterized in that: The image processing device appends the detection results to the captured image by combining the detection results acquired at a time corresponding to the captured image as an image, and encodes the captured image containing the detection results before sending it to the display device. The control device obtains the detection result from the captured image decoded by the display device.

6. The remote control system according to any one of claims 1 to 5, characterized in that: The control device, acting as the action control, controls the action of the slave device and the action of the master device based on the user's operation of the master device and the detection results, so as to inform the user of the reaction force acting on the slave device.

7. The remote control system according to claim 6, characterized in that: The sensor is a force sensor.

8. A control method for a remote control system, comprising a master device, a slave device, a sensor, a camera device, and a display device, wherein the master device is operated by a user, the slave device applies actions to an object according to the actions of the master device, the sensor is disposed on the slave device to detect the operating state of the slave device, the camera device captures images of at least one of the slave device and the object, and the display device displays the captured images and provides them to the user operating the master device, characterized in that: The control method of the remote control system includes: The steps of executing motion control of at least one of the master device and the slave device based on the detection results of the sensor; and The step of delaying the motion control to reduce the deviation in display timing of the captured image caused by the display device for the motion control includes receiving the detection result from the slave device and receiving the captured image from the display device. The delay amount for delaying the motion control is determined based on the deviation time between the detection result and the captured image by aligning the signal waveform of the captured image with the corresponding signal waveform of the detection result.

9. A remote control program for enabling a computer to control a remote control system, the remote control system comprising a master device, slave devices, sensors, a camera device, and a display device, wherein the master device is operated by a user, the slave devices apply actions to an object according to the actions of the master device, the sensors are disposed on the slave devices to detect the operating state of the slave devices, the camera device captures images of at least one of the slave devices and the object, and the display device displays the captured images and provides them to the user operating the master device, characterized in that: The remote control program enables the computer to: Based on the detection results of the sensors, the system performs motion control functions for at least one of the master device and the slave device; and The function of delaying the motion control to reduce the timing deviation of the display of the captured image caused by the display device for the motion control, wherein the detection result is received from the slave device and the captured image is received from the display device, and the delay amount of delaying the motion control is determined based on the time deviation between the detection result and the captured image by aligning the signal waveform of the captured image with the corresponding signal waveform of the detection result.

Images