Remote control system, control method for remote control system, and remote control program

Abstract

To provide a remote control system accompanying display of a photographic image which reduces discomfort given to a user.SOLUTION: A remote control system 100 includes: an operation device 2 operated by a user; a robot 1 for applying an action to an object W according to the operation of the operation device 2; a contact force sensor 13 that is provided on the robot 1 and detects the operation state of the robot 1; an imaging device 71 for imaging at least one image of the robot 1 and the object W; a display device 8 for displaying a photographic image by the imaging device 71, and providing the displayed photographic image to a user operating the operation device 2; and a control device 3 for executing at least one operation control of the robot 1 and the operation device 2 on the basis of a detection result of the contact force sensor 13. The control device 3 delays the operation control so as to reduce a deviation of display timing by the display device 8 of the photographic image with respect to the operation control.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technology disclosed herein relates to a remote control system, a control method for the remote control system, and a remote control program.

Background Art

[0002] Conventionally, a technology for remotely controlling a slave device by a master device has been known. For example, Patent Document 1 discloses a remote control system including a master device and a slave device arranged at physically separated positions. In this remote control system, a display device for displaying an image transmitted from the slave device side to the master device side is provided. When the communication delay between the master device and the slave device is large, the image is displayed on the display device in a blurred manner. Thereby, the user can know that the communication delay is large.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As described above, in a remote control system, not only control signals but also various signals such as image signals can be transmitted and received between the master device and the slave device. The time required for signal transmission and reception and the time required for signal processing vary for each signal. If a time deviation for each signal occurs in the final signal processing, there is a risk of giving the user a sense of discomfort. For example, in the case of the above-described remote control system, if a time deviation occurs between the control between the master device and the slave device and the image display on the display device, the user may feel a sense of discomfort.

[0005] The technology disclosed herein has been made in view of such points, and its object is to reduce the discomfort given to a user in a remote control system involving display of a photographed image.

Means for Solving the Problem

[0006] The remote control system disclosed herein includes a master device operated by a user, a slave device that applies an action to an object according to the operation of the master device, a sensor provided in the slave device for detecting the operating state of the slave device, an imaging device that captures an image of at least one of the slave device and the object, a display device that displays the photographed image by the imaging device and provides it to the user operating the master device, and a control device that executes operation control of at least one of the master device and the slave device based on the detection result of the sensor. The control device delays the operation control so as to reduce the deviation in the display timing of the photographed image by the display device with respect to the operation control.

[0007] The control method of the remote control system disclosed herein is a control method of a remote control system including a master device operated by a user, a slave device that applies an action to an object according to the operation of the master device, a sensor provided in the slave device for detecting the operating state of the slave device, an imaging device that captures an image of at least one of the slave device and the object, and a display device that displays the photographed image by the imaging device and provides it to the user operating the master device. The method includes executing operation control of at least one of the master device and the slave device based on the detection result of the sensor, and delaying the operation control so as to reduce the deviation in the display timing of the photographed image by the display device with respect to the operation control.

[0008] The remote control program disclosed herein causes a computer to realize a function for controlling a remote control system including a master device operated by a user, a slave device that applies an action to an object according to the operation of the master device, a sensor provided in the slave device for detecting an operating state of the slave device, an imaging device that captures an image of at least one of the slave device and the object, and a display device that displays a captured image by the imaging device and provides it to the user who operates the master device. The computer is caused to realize a function of performing operation control of at least one of the master device and the slave device based on a detection result of the sensor, and a function of delaying the operation control so as to reduce a deviation in display timing of the captured image by the display device with respect to the operation control.

Advantages of the Invention

[0009] According to the remote control system, it is possible to reduce the sense of discomfort given to the user in a remote control system involving display of a captured image.

[0010] According to the control method of the remote control system, it is possible to reduce the sense of discomfort given to the user in a remote control system involving display of a captured image.

[0011] According to the remote control program, it is possible to reduce the sense of discomfort given to the user in a remote control system involving display of a captured image.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0013] Hereinafter, exemplary embodiments will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram showing a configuration of a remote control system 100 according to the embodiment. Note that the broken lines in FIG. 1 represent wireless communication.

[0014] The remote control system 100 includes a master-slave system 110. The master-slave system 110 has an operating device 2 operated by a user, a robot 1 that applies an action to an object W according to the operation of the operating device 2, and a control device 3 that controls the robot 1 and the operating device 2. The operating device 2 and the control device 3 are arranged at the first site S1. The robot 1 is arranged at a second site S2 different from the first site S1. The master-slave system 110 realizes remote control between the operating device 2 and the robot 1. The operating device 2 is an example of a master device, and the robot 1 is an example of a slave device.

[0015] In addition, in the present disclosure, the work performed by the robot 1 does not include teaching work and confirmation and correction work of teaching. Therefore, the operating device 2 does not include a teach pendant.

[0016] The robot 1 and the operating device 2 are communicably connected. Specifically, the robot 1 and the control device 3 are wirelessly communicably connected. That is, signals are transmitted and received between the robot 1 and the control device 3 via the transmission path of the robot 1, wireless communication, and the control device 3 (hereinafter referred to as the "first transmission path"). The operating device 2 is connected to the control device 3. That is, the operating device 2 communicates with the robot 1 via the control device 3.

[0017] In this example, the robot 1 is an industrial robot. The robot 1 applies an action to the object W. The action is specifically processing, and more specifically grinding. The action is not limited to grinding, and may be grinding or polishing, etc. The object W is, for example, a large steel plate or the wall of a large tank.

[0018] The robot 1 has a sensor that detects the operating state of the robot 1. In this example, the sensor further has a contact force sensor 13 that detects the reaction force (hereinafter referred to as the "contact force") received by the robot 1 from the object W.

[0019] The control device 3 receives the detection result of the contact force sensor 13 via the robot 1. The control device 3 executes operation control of at least one of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13. In this example, the control device 3 controls the operation of the robot 1 and controls the operation of the operating device 2 so as to present the reaction force acting on the robot 1 to the user according to the operation of the operating device 2 by the user and the detection result of the contact force sensor 13.

[0020] The remote control system 100 further includes an image system 120. The image system 120 includes an imaging device 71 that captures an image, and a display device 8 that displays the captured image by the imaging device 71. The image system 120 further includes an image processing device 72 that processes the captured image of the imaging device 71. The display device 8 is disposed at the first site S1. The display device 8 is disposed at a position where the user can visually recognize while operating the operating device 2. The imaging device 71 and the image processing device 72 are disposed at the second site S2. The imaging device 71 captures an image of the robot 1 and / or the object W.

[0021] The imaging device 71 and the display device 8 are communicably connected. Specifically, the image processing device 72 and the display device 8 are wirelessly communicably connected. That is, signals are transmitted and received between the imaging device 71 and the display device 8 via the transmission path of the image processing device 72, wireless communication, and the display device 8 (hereinafter referred to as the "second transmission path"). The imaging device 71 is connected to the image processing device 72. That is, the imaging device 71 communicates with the display device 8 via the image processing device 72.

[0022] The image system 120 assists the user operating the operating device 2 by capturing an image of the robot 1 and / or the object W with the imaging device 71 and displaying the captured image on the display device 8.

[0023] <Detailed Configuration of Remote Control System>

[0024] [Robot] Robot 1 may have an end effector 11 that acts on the object W and a robot arm 12 that operates the end effector 11. Robot 1 operates, i.e., moves, the end effector 11 by the robot arm 12 and acts on the object W by the end effector 11. Robot 1 may further have a base 10 that supports the robot arm 12 and a robot control device 14 that controls the entire robot 1.

[0025] A robot coordinate system of three orthogonal axes is defined for robot 1. For example, the Z axis is set in the vertical direction, and the X axis and the Y axis that are orthogonal to each other in the horizontal direction are set.

[0026] The end effector 11 has a grinding device 11a and applies grinding as an action to the object W. For example, the grinding device 11a may be a grinder, an orbital sander, a random orbital sander, a delta sander, a belt sander, or the like. The grinder may be of a type that rotates a disk-shaped grinding wheel, a type that rotates a conical or cylindrical grinding wheel, or the like. Here, the grinding device 11a is a grinder.

[0027] The robot arm 12 changes the position of the grinding device 11a. Further, the robot arm 12 may change the posture of the grinding device 11a. The robot arm 12 is a vertically articulated robot arm. The robot arm 12 has a plurality of links 12a, joints 12b that connect the plurality of links 12a, and servo motors 15 (see FIG. 2) that rotationally drive the plurality of joints 12b.

[0028] Incidentally, the robot arm 12 may be a horizontally articulated type, a parallel link type, a rectangular coordinate type, a polar coordinate type robot arm, or the like.

[0029] In this example, the contact force sensor 13 is provided between the robot arm 12 and the end effector 11 (specifically, at the connection between the robot arm 12 and the end effector 11). The contact force sensor 13 detects the contact force that the end effector 11 receives from the object W. The contact force sensor 13 detects forces in three orthogonal axial directions and moments about these three axes. The contact force sensor 13 is an example of a force sensor.

[0030] Note that the force sensor is not limited to the contact force sensor 13. For example, the contact force sensor 13 may detect only forces in one, two, or three axial directions. Alternatively, the force sensor may be a current sensor that detects the current of the servo motor 15 of the robot arm 12 or a torque sensor that detects the torque of the servo motor 15, etc.

[0031] FIG. 2 is a diagram showing a schematic hardware configuration of the robot control device 14. The robot control device 14 controls the servo motor 15 of the robot arm 12 and the grinding device 11a. The robot control device 14 receives the detection signal of the contact force sensor 13. The robot control device 14 transmits and receives information, commands, data, etc. to and from the control device 3 and the image processing device 72. The robot control device 14 performs wireless communication with the control device 3. The robot control device 14 is connected to the image processing device 72 by a wiring and transmits information and data, etc. to the image processing device 72 via the wiring. The robot control device 14 includes a control unit 16, a storage unit 17, a memory 18, and a communication unit 19.

[0032] The control unit 16 controls the entire robot control device 14. The control unit 16 performs various arithmetic processes. For example, the control unit 16 is formed of a processor such as a CPU (Central Processing Unit). The control unit 16 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, etc.

[0033] The memory unit 17 stores the programs and various data executed by the control unit 16. The memory unit 17 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0034] The memory 18 temporarily stores data and the like. For example, the memory 18 is formed of a volatile memory.

[0035] The communication unit 19 is a communication module and executes wireless communication. For example, the communication unit 19 realizes wireless communication in accordance with the 5G communication standard.

[0036] [Operating device] As shown in FIG. 1, the operating device 2 includes an operation unit 21 operated by a user and an operation force sensor 23 that detects an operation force applied by the user to the operation unit 21. The operating device 2 receives an input for manually operating the robot 1 and outputs operation information, which is the input information, to the control device 3. Specifically, the user holds and operates the operation unit 21. At this time, the operation force sensor 23 detects the force applied to the operation unit 21. The operation force detected by the operation force sensor 23 is output to the control device 3 as operation information.

[0037] The operating device 2 may further include a base 20, a support mechanism 22 provided on the base 20 and supporting the operation unit 21, and an operation control device 24 that controls the entire operating device 2. The operating device 2 presents a reaction force against the operation force to the user under the control from the control device 3. Specifically, the operation control device 24 receives a command from the control device 3 and controls the support mechanism 22 to make the user sense the reaction force.

[0038] An orthogonal three-axis operation coordinate system is defined for the operating device 2. The operation coordinate system corresponds to the robot coordinate system. That is, the Z axis is set in the vertical direction, and the X axis and the Y axis perpendicular to each other are set in the horizontal direction.

[0039] The support mechanism 22 includes a plurality of links 22a, joints 22b that connect the plurality of links 22a, and a servo motor 25 (see FIG. 3) that rotationally drives the plurality of joints 22b. The support mechanism 22 supports the operation unit 21 so that the operation unit 21 can assume an arbitrary position and posture in a three-dimensional space. The servo motor 25 rotates corresponding to the position and posture of the operation unit 21. The amount of rotation of the servo motor 25, that is, the rotation angle, is uniquely determined.

[0040] In this example, the operation force sensor 23 is provided between the operation unit 21 and the support mechanism 22 (specifically, at the connection part between the operation unit 21 and the support mechanism 22). The operation force sensor 23 detects forces in three orthogonal axial directions and moments about the three axes.

[0041] Note that the detection unit for the operation force is not limited to the operation force sensor 23. For example, the operation force sensor 23 may detect only forces in one-axis, two-axis, or three-axis directions. Alternatively, the detection unit may be a current sensor that detects the current of the servo motor 25 of the support mechanism 22 or a torque sensor that detects the torque of the servo motor 25, etc.

[0042] FIG. 3 is a diagram showing a schematic hardware configuration of the operation control device 24. The operation control device 24 operates the support mechanism 22 by controlling the servo motor 25. The operation control device 24 receives the detection signal of the operation force sensor 23. The operation control device 24 is connected to the control device 3 by wiring and performs transmission and reception of information, commands, data, etc. with the control device 3 via the wiring. The operation control device 24 includes a control unit 26, a storage unit 27, and a memory 28.

[0043] The control unit 26 controls the entire operation control device 24. The control unit 26 performs various arithmetic processes. For example, the control unit 26 is formed of a processor such as a CPU (Central Processing Unit). The control unit 26 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0044] The storage unit 27 stores the programs and various data executed by the control unit 26. The storage unit 27 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0045] The memory 28 temporarily stores data and the like. For example, the memory 28 is formed of a volatile memory.

[0046] [Control Device] The control device 3 controls the entire master-slave system 110 and controls the operations of the robot 1 and the operating device 2. Specifically, the control device 3 performs master-slave control, specifically, bilateral control, between the robot 1 and the operating device 2. The control device 3 executes a first control for controlling the operation of the robot 1 according to the operation of the operating device 2 by the user's operation, and a second control for controlling the operation of the operating device 2 so as to present the reaction force according to the detection result of the contact force sensor 13 to the user. That is, the end effector 11 processes the object W by the first control, and the reaction force during processing is presented to the user by the second control.

[0047] FIG. 4 is a diagram showing a schematic hardware configuration of the control device 3. The control device 3 transmits and receives information, commands, data, etc. to and from the robot control device 14 and the operation control device 24. Further, the control device 3 transmits and receives information, commands, data, etc. to and from the display device 8. The control device 3 includes a control unit 31, a storage unit 32, a memory 33, and a communication unit 34. Although not shown, the control device 3 may further include an input operation unit operated by the user to set the operation control of the robot 1 and the operation device 2, and a display for displaying the set content.

[0048] The control unit 31 controls the entire control device 3. The control unit 31 performs various arithmetic processes. For example, the control unit 31 is formed of a processor such as a CPU (Central Processing Unit). The control unit 31 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0049] The storage unit 32 stores programs and various data executed by the control unit 31. For example, the storage unit 32 stores a remote control program for controlling the remote control system 100. The storage unit 32 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0050] The memory 33 temporarily stores data and the like. For example, the memory 33 is formed of a volatile memory.

[0051] The communication unit 34 is a communication module that performs wireless communication. For example, the communication unit 34 realizes wireless communication in accordance with the 5G communication standard.

[0052] [Imaging Device and Image Processing Device] The imaging device 71 captures images, specifically videos, of the robot 1 and the object W. In this example, since the robot 1 grinds the object W, the imaging device 71 captures images centered on the end effector 11 and the object W. The imaging device 71 captures videos at a predetermined frame rate. The captured images of the imaging device 71 are input to the image processing device 72 as image signals.

[0053] The image processing device 72 receives the captured images of the imaging device 71, that is, the image signals, processes the image signals, and transmits the processed image signals to the display device 8. The image processing device 72 encodes the processed image signals and transmits the encoded image signals to the display device 8 via wireless communication.

[0054] Specifically, in addition to the image signals of the imaging device 71, the image processing device 72 receives the detection results of the contact force sensor 13 from the robot 1. As processing of the captured images, the image processing device 72 associates the captured images of the imaging device 71 and the detection results of the contact force sensor 13 acquired at corresponding timings. Specifically, the image processing device 72 adds the detection results of the contact force sensor 13 acquired at the timing corresponding to the captured image to the captured image. The image processing device 72 encodes the captured image with the detection results of the contact force sensor 13 added thereto and transmits it to the display device 8.

[0055] FIG. 5 is a diagram showing a schematic hardware configuration of the image processing device 72. The image processing device 72 includes a control unit 73, a storage unit 74, a memory 75, and a communication unit 76.

[0056] The control unit 73 controls the entire image processing device 72. The control unit 73 performs various arithmetic processes. For example, the control unit 73 is formed of a processor such as a CPU (Central Processing Unit). The control unit 73 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0057] The storage unit 74 stores the programs and various data executed by the control unit 73. The storage unit 74 is formed of a non-volatile memory, a HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0058] The memory 75 temporarily stores data and the like. For example, the memory 75 is formed of a volatile memory.

[0059] The communication unit 76 is a communication module and performs wireless communication. For example, the communication unit 76 realizes wireless communication in accordance with the 5G communication standard.

[0060] [Display device] The display device 8 displays the captured image of the imaging device 71. The display device 8 includes a display 81 and a display control device 82.

[0061] The display control device 82 receives the captured image of the imaging device 71 by wireless communication. Specifically, the display control device 82 performs wireless communication with the image processing device 72. The display control device 82 receives an image signal from the image processing device 72. The display control device 82 decodes the received image signal and outputs the decoded image signal to the display 81.

[0062] The display 8 displays the image signal input from the display control device 82 as an image.

[0063] FIG. 6 is a diagram showing a schematic hardware configuration of the display control device 82. The display control device 82 includes a control unit 83, a storage unit 84, a memory 85, and a communication unit 86.

[0064] The control unit 83 controls the entire display control device 82. The control unit 83 performs various arithmetic processes. For example, the control unit 83 is formed of a processor such as a CPU (Central Processing Unit). The control unit 83 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0065] The storage unit 84 stores programs and various data executed by the control unit 83. The storage unit 84 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0066] The memory 85 temporarily stores data and the like. For example, the memory 85 is formed of a volatile memory.

[0067] The communication unit 86 is a communication module and performs wireless communication. For example, the communication unit 86 realizes wireless communication conforming to the 5G communication standard.

[0068] <Control of Remote Control System> In the remote control system 100 configured as described above, the control device 3 performs master-slave control in parallel, which includes first control for controlling the operation of the robot 1 according to the operation of the operation device 2 by the user, and second control for controlling the operation of the operation device 2 so as to present a reaction force according to the detection result of the contact force sensor 13 to the user. In parallel with this, the image system 120 performs image display control for photographing the robot 1 and the object W by the imaging device 71 and displaying the photographed image by the display device 8. The user can operate the operation device 2 while viewing the photographed image displayed on the display device 8. At this time, the control device 3 executes delay control for delaying the master-slave control so as to reduce the deviation between the master-slave control and the photographed image displayed on the display device 8.

[0069] Furthermore, while executing master-slave control and image display control, the control device 3 obtains the deviation time between the master-slave control and the captured image displayed on the display device 8, and updates the delay amount of the delay control based on the deviation time.

[0070] FIG. 7 is a block diagram showing the configuration of the control system of the remote control system 100.

[0071] The control unit 16 of the robot control device 14 realizes various functions by reading out and expanding a program from the storage unit 17 to the memory 18. Specifically, the control unit 16 functions as an input processing unit 41 and an operation control unit 42.

[0072] The input processing unit 41 outputs information, data, commands, etc. received from the contact force sensor 13 and the servo motor 15 to the control device 3. Specifically, the input processing unit 41 receives a six-axis force detection signal (hereinafter referred to as "sensor signal") from the contact force sensor 13, and outputs the sensor signal to the control device 3. The sensor signal is transmitted to the control device 3 by wireless communication. In addition, the input processing unit 41 receives detection signals from a rotation sensor (for example, an encoder) and a current sensor from the servo motor 15. The input processing unit 41 outputs the detection signals to the operation control unit 42 for feedback control of the robot arm 12 by the operation control unit 42. In addition, the input processing unit 41 outputs the detection signals to the control device 3 as the position information of the robot arm 12. In addition, the input processing unit 41 also outputs the sensor signal of the contact force sensor 13 to the image processing device 72.

[0073] The motion control unit 42 receives the commanded position xds from the control device 3 via wireless communication, and generates a control command for operating the robot arm 12 according to the commanded position xds. The motion control unit 42 outputs the control command to the servo motor 15, operates the robot arm 12, and moves the grinding device 11a to the position corresponding to the commanded position. At this time, the motion control unit 42 performs feedback control on the operation of the robot arm 12 based on the detection signals of the rotation sensor and / or current sensor of the servo motor 15 from the input processing unit 41. In addition, the motion control unit 42 outputs a control command to the grinding device 11a and operates the grinding device 11a. Thereby, the grinding device 11a grinds the object W.

[0074] The control unit 26 of the operation control device 24 realizes various functions by reading and expanding a program from the storage unit 27 into the memory 28. Specifically, the control unit 26 functions as an input processing unit 51 and a motion control unit 52.

[0075] The input processing unit 51 outputs information, data, commands, etc. received from the operation force sensor 23 to the control device 3. Specifically, the input processing unit 51 receives the detection signals of the six-axis force from the operation force sensor 23 and outputs the detection signals to the control device 3. In addition, the input processing unit 51 receives the detection signals of the rotation sensor (for example, an encoder) and the current sensor from the servo motor 25. The input processing unit 51 outputs the detection signals to the motion control unit 52 for the feedback control of the support mechanism 22 by the motion control unit 52.

[0076] The operation control unit 52 receives the command position xdm from the control device 3 and generates a control command for operating the support mechanism 22 according to the command position xdm. The operation control unit 52 outputs the control command to the servo motor 25, operates the support mechanism 22, and moves the operation unit 21 to a position corresponding to the command position. At this time, the operation control unit 52 performs feedback control on the operation of the support mechanism 22 based on the detection signals of the rotation sensor and / or current sensor of the servo motor 25 from the input processing unit 51. Thereby, a reaction force is applied to the operation force applied by the user to the operation unit 21. As a result, the user can operate the operation unit 21 while pseudo-feeling the reaction force from the object W from the operation unit 21.

[0077] The control unit 31 of the control device 3 realizes various functions by reading a program (specifically, a remote control program) from the storage unit 32 and expanding it into the memory 33. Specifically, the control unit 31 functions as an operation force acquisition unit 61, a contact force acquisition unit 62, an addition unit 63, a force / velocity conversion unit 64, a first velocity / position conversion unit 65, a second velocity / position conversion unit 66, a separation unit 67, a deviation detection unit 68, and a delay unit 69.

[0078] Also, the control unit 73 of the image processing device 72 realizes various functions by reading a program from the storage unit 74 and expanding it into the memory 75. Specifically, the control unit 73 functions as a synthesis unit 77 and an encoder 78.

[0079] The control unit 83 of the display control device 82 realizes various functions by reading a program from the storage unit 84 and expanding it into the memory 85. Specifically, the control unit 83 functions as a decoder 87.

[0080] The operation force acquisition unit 61 receives the detection signal of the operation force sensor 23 via the input processing unit 51 and acquires the operation force fm based on the detection signal. The operation force acquisition unit 61 inputs the operation force fm to the addition unit 63.

[0081] The contact force acquisition unit 62 receives the sensor signal of the contact force sensor 13 via the input processing unit 41, and acquires the contact force fs based on the sensor signal. The contact force acquisition unit 62 inputs the contact force fs to the addition unit 63. In addition, the contact force acquisition unit 62 outputs the sensor signal to the deviation detection unit 68.

[0082] The addition unit 63 calculates the sum of the operating force fm input from the operating force acquisition unit 61 and the contact force fs input from the contact force acquisition unit 62. Here, since the operating force fm and the contact force fs are forces in opposite directions, the operating force fm and the contact force fs have different positive and negative signs. That is, by adding the operating force fm and the contact force fs, the absolute value of the operating force fm becomes smaller. The addition unit 63 outputs the combined force fm + fs, which is the sum of the operating force fm and the contact force fs.

[0083] The delay unit 69 has a function of delaying the master-slave control. In this example, the delay unit 69 is a low-pass filter, and more specifically, a first-order lag filter. The first-order lag filter includes a time constant. The delay unit 69 delays the input of the combined force fm + fs from the addition unit 63 to the force / velocity conversion unit 64 according to the time constant.

[0084] The force / velocity conversion unit 64 converts the combined force fm + fs input via the delay unit 69 into the commanded velocity xd’. The force / velocity conversion unit 64 calculates the commanded velocity xd’ using a motion model based on the equation of motion including the inertia coefficient, the viscous coefficient (damping coefficient), and the stiffness coefficient (spring coefficient). Specifically, the force / velocity conversion unit 64 calculates the commanded velocity xd’ based on the following equation of motion.

[0085]

Equation

[0086] Equation (1) is a linear differential equation. Solving Equation (1) for xd’ results in Equation (2).

[0087]

Number

[0088] Equation (2) is stored in the memory unit 32. The force / velocity conversion unit 64 reads Equation (2) from the memory unit 32 to obtain the commanded velocity xd’, and outputs the obtained commanded velocity xd’ to the first velocity / position conversion unit 65 and the second velocity / position conversion unit 66.

[0089] The first velocity / position conversion unit 65 converts the coordinate-transformed commanded velocity xd’ into a commanded position xds for the robot 1 based on the robot coordinate system. For example, when the ratio of the movement amount of the robot 1 to the movement amount of the operating device 2 is set, the first velocity / position conversion unit 65 multiplies the commanded position xd obtained from the commanded velocity xd’ by the movement ratio to obtain the commanded position xds. The first velocity / position conversion unit 65 outputs the obtained commanded position xds to the robot control device 14, specifically, the motion control unit 42. The motion control unit 42 operates the robot arm 12 based on the commanded position xds as described above.

[0090] The second velocity / position conversion unit 66 converts the commanded velocity xd’ into a commanded position xdm for the operating device 2 based on the operation coordinate system. The second velocity / position conversion unit 66 outputs the obtained commanded position xdm to the operation control device 24, specifically, the motion control unit 52. The motion control unit 52 operates the support mechanism 22 based on the commanded position xdm as described above.

[0091] The synthesizing unit 77 adds the detection result of the contact force sensor 13, which is acquired at a timing corresponding to the captured image of the imaging device 71, to the captured image of the imaging device 71 as association information indicating the association between them. Specifically, the synthesizing unit 77 receives an image signal from the imaging device 71 and also receives the sensor signal of the contact force sensor 13 via the input processing unit 41. The synthesizing unit 77 synthesizes the sensor signal with the image signal. The synthesizing unit 77 synthesizes the sensor signal and the image signal that are acquired at timings corresponding to each other (i.e., substantially the same timing) by the contact force sensor 13 and the imaging device 71. For example, the synthesizing unit 77 regards the time when the image signal and the sensor signal are respectively received by the synthesizing unit 77 as the acquisition time of the image signal and the sensor signal respectively. In this way, the sensor signal acquired at a timing corresponding to the image signal is added to the image signal as association information indicating the association between them.

[0092] The sensor signal includes sensor signals of six-axis forces. The synthesizing unit 77 synthesizes at least one of the sensor signals of the six-axis forces with the image signal. Preferably, the synthesizing unit 77 synthesizes, with the image signal, the sensor signal with a large variation when the robot 1 acts on the object W among the sensor signals of the six-axis forces. In this example, since the action on the object W is grinding, the synthesizing unit 77 synthesizes the sensor signal of the force in the Z-axis direction with the image signal.

[0093] At this time, the synthesizing unit 77 synthesizes the sensor signal with the image signal by using an electronic watermark technique. Preferably, the synthesizing unit 77 embeds the sensor signal into the image signal by a perception-difficult type of electronic watermark.

[0094] The encoder 78 encodes the image signal in which the sensor signal is synthesized by the synthesizing unit 77. The encoder 78 outputs the encoded image signal to the communication unit 76. The encoded image signal is transmitted to the display control device 82 via the communication unit 76.

[0095] The display control device 82 receives an image signal from the image processing device 72 via the communication unit 86. The image signal is input to the decoder 87. The decoder 87 decodes the image signal. The decoder 87 outputs the decoded image signal to the display 81 and the separation unit 67 of the control device 3.

[0096] The display 81 displays an image according to the decoded image signal.

[0097] The separation unit 67 separates the sensor signal from the decoded image signal. In this example, the sensor signal is synthesized into the image signal by digital watermarking. The separation unit 67 separates the sensor signal from the image signal by digital watermarking technology. The separation unit 67 outputs the separated sensor signal to the deviation detection unit 68.

[0098] The deviation detection unit 68 obtains and updates the delay time as the delay amount for delaying the master-slave control. The deviation detection unit 68 obtains the delay time based on the deviation time between the sensor signal received from the robot 1 and the image signal received from the display device 8. Here, the deviation time between the sensor signal and the image signal is originally the timing corresponding to each other, that is, the deviation in the time when the sensor signal and the image signal acquired at substantially the same timing are received by the deviation detection unit 68, that is, the control device 3. Even if the original acquisition timings of the sensor signal and the image signal are substantially the same, a deviation occurs in the time until they reach the control device 3 because the transmission paths to the control device 3 and the processes received in the middle are different.

[0099] In addition, the sensor signal received from the robot 1 is used to derive the command positions xds and xdm. Also, the image signal received from the display device 8 is the same signal as the image signal input to the display 81. That is, the deviation time between the sensor signal received from the robot 1 and the image signal received from the display device 8 corresponds to the deviation in the display timing of the captured image by the display device 8 for the master-slave control.

[0100] Specifically, the deviation detection unit 68 obtains the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal from the separation unit 67. The deviation detection unit 68 receives the sensor signal from the contact force acquisition unit 62 and also receives the sensor signal from the separation unit 67. The sensor signal from the contact force acquisition unit 62 is the sensor signal received from the robot control device 14, that is, the robot 1. On the other hand, the sensor signal from the separation unit 67 is a signal separated from the image signal decoded in the display device 8, and is a signal acquired by the contact force sensor 13 at a timing corresponding to the timing when the image signal was acquired by the imaging device 71. Therefore, the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal from the separation unit 67 corresponds to the deviation time between the sensor signal received from the robot 1 and the image signal received from the display device 8.

[0101] In addition, since the sensor signal synthesized into the image signal by the synthesis unit 77 is the sensor signal of the force in the Z-axis direction with large fluctuations during grinding among the six-axis force sensor signals, the deviation detection unit 68 compares the sensor signal of the force in the Z-axis direction among the sensor signals from the contact force acquisition unit 62 with the sensor signal from the separation unit 67.

[0102] The deviation detection unit 68 stores the obtained deviation time in the storage unit 32. The deviation time accumulates in the storage unit 32.

[0103] The storage unit 32 stores a delay time that serves as a reference for determining the time constant of the delay unit 69. The initial value of the delay time is set by obtaining the deviation time in advance as described above. The deviation detection unit 68 updates the delay time based on the deviation time accumulated in the storage unit 32. The deviation detection unit 68 updates the delay time with the deviation time when a state where the deviation time is significantly different from the delay time continues for a certain period or more.

[0104] When the delay time is updated, the deviation detection unit 68 obtains the time constant to be set in the delay unit 69 based on the delay time. Specifically, a conversion function for converting the delay time into a time constant is stored in the storage unit 32 in the storage unit 32. The deviation detection unit 68 obtains the time constant from the delay time using the conversion function. The deviation detection unit 68 updates the time constant of the delay unit 69 with the obtained time constant. The delay unit 69 executes the above-described process using the updated time constant.

[0105] Next, the operation of the remote control system 100 configured as described above will be described. FIG. 8 is a flowchart showing the operation of the remote control system 100.

[0106] [Operation of Master-Slave System] The master-slave control by the master-slave system 110 will be described. The master-slave control is realized by the control device 3 reading out and expanding the remote control program stored in the storage unit 32 into the memory 33.

[0107] First, in step Sa0, delay control is set. The control device 3 displays a setting screen for setting delay control on a display (not shown). Specifically, the display shows a display for switching the validity of delay control, the delay amount (specifically, the delay time) of the master-slave control when delay control is enabled, and the like. The user sets the delay control via the input operation unit.

[0108] Subsequently, in step Sa1, when the user operates the operating device 2, the operating force sensor 23 detects the operating force applied by the user via the operating unit 21. At this time, the contact force sensor 13 of the robot 1 detects the contact force.

[0109] The operating force detected by the operating force sensor 23 is input to the control device 3 as a detection signal by the input processing unit 51. In the control device 3, the operating force acquisition unit 61 inputs the operating force fm based on the detection signal to the addition unit 63.

[0110] The contact force detected by the contact force sensor 13 is input to the input processing unit 41 as a sensor signal. The sensor signal input to the input processing unit 41 is transmitted to the control device 3 via the communication unit 19. In addition, the input processing unit 41 inputs the sensor signal to the image processing device 72.

[0111] In the control device 3, the communication unit 34 receives the sensor signal and inputs the sensor signal to the contact force acquisition unit 62. The contact force acquisition unit 62 inputs the contact force fs based on the sensor signal to the addition unit 63. In addition, the contact force acquisition unit 62 inputs the sensor signal to the deviation detection unit 68. The deviation detection unit 68 stores the sensor signal in the memory 33.

[0112] Next, in step Sa2, the control device 3 determines whether the delay control is effective.

[0113] When the delay control is invalid, in step Sa4, the grinding process by the robot 1 and the presentation of the reaction force by the operating device 2 are executed. Specifically, the addition unit 63 obtains the combined force fm + fs based on the input operating force fm and contact force fs, and inputs the combined force fm + fs to the force / velocity conversion unit 64. At this time, the delay unit 69 is treated as invalid. That is, the combined force fm + fs output from the addition unit 63 is input to the force / velocity conversion unit 64 without delay. The force / velocity conversion unit 64 obtains the command velocity xd' based on Equation (2) using the combined force fm + fs.

[0114] Regarding the robot 1, the first velocity / position conversion unit 65 obtains the command position xds from the command velocity xd'. The operation control unit 42 of the robot control device 14 operates the robot arm 12 according to the command position xds and controls the position of the grinding device 11a. Thereby, while the pressing force corresponding to the operating force fm is applied to the object W, the object W is ground by the grinding device 11a.

[0115] Regarding the operating device 2, the second speed / position conversion unit 66 obtains a command position xdm from the command speed xd’. The operation control unit 52 of the operation control device 24 operates the support mechanism 22 according to the command position xdm to control the position of the operation unit 21. Thereby, the user senses a reaction force corresponding to the contact force fs.

[0116] On the other hand, when the delay control is effective, in step Sa3, the master-slave control is delayed. Specifically, the delay unit 69 delays the input of the combined force fm + fs from the addition unit 63 to the force / speed conversion unit 64 based on the set time constant. As a result, the time from when the detection signal of the operation force sensor 23 and the sensor signal of the contact force sensor 13 are input until the operation control of the robot arm 12 and the operation unit 21 is executed is delayed.

[0117] The master-slave system 110 repeats such processing in a predetermined control cycle. The user can remotely control the robot 1 arranged at the second site S2 to grind the object W by operating the operating device 2 at the first site S1. At that time, the user can operate the operating device 2 while sensing the reaction force received by the grinding device 11a from the object W via the operating device 2. Note that step Sa0 is executed only once at the start of this control and is not executed in subsequent control cycles.

[0118] [Operation of the Image System] In parallel with this, the image system 120 executes image display control for photographing the robot 1 and the object W and presenting the photographed image to the user.

[0119] First, in step Sb1, the imaging device 71 photographs the robot 1 and the object W. The imaging device 71 photographs a moving image of the robot 1 and the object W at a predetermined frame rate. The imaging device 71 inputs the photographed image (i.e., the image signal) to the image processing device 72.

[0120] In step Sb2, the synthesizing unit 77 of the image processing apparatus 72 synthesizes the sensor signal of the contact force sensor 13 with the captured image. This synthesis is not a process for presenting the captured image to the user, but a process performed to update the delay time of the master-slave control.

[0121] Specifically, FIG. 9(i) is an example of the sensor signal output from the input processing unit 41. FIG. 9(ii) is an example of the image signal output from the imaging device 71. FIG. 10 is an example of the image signal in which the sensor signal is synthesized, output from the synthesizing unit 77.

[0122] As shown in FIG. 9(i), the sensor signal output from the input processing unit 41 includes data acquired at a sampling period corresponding to the control period of the master-slave control. On the other hand, as shown in FIG. 9(ii), the image signal input from the imaging device 71 includes frames (still images) acquired at a predetermined frame rate. In this example, the sampling period is shorter than the frame rate. Therefore, the synthesizing unit 77 synthesizes the sensor signal acquired between each frame and the previous frame with each frame. The synthesizing unit 77 synthesizes the sensor signal as an image with the image of each frame, that is, synthesizes the signal waveforms in time series. The synthesis of the sensor signal is performed by the electronic watermark technology. As a result, as shown in FIG. 10, an image signal in which the image of the sensor signal is synthesized as an electronic watermark with the frame is generated. The synthesized frame and the sensor signal are acquired at substantially the same timing.

[0123] In step Sb3, the encoder 78 encodes the image signal generated by the synthesizing unit 77.

[0124] Thereafter, in step Sb4, the image processing apparatus 72 transmits the encoded image signal to the display control device 82 by wireless communication. The display control device 82 receives the image signal from the image processing apparatus 72.

[0125] In the control device 82, the decoder 87 decodes the image signal in step Sb5. The decoder 87 outputs the decoded image signal to the display 81.

[0126] In step Sb6, the display 81 displays an image based on the image signal. Thereby, the captured image by the imaging device 71 is presented to the user. Here, the display 81 displays the captured image in which the sensor signal is synthesized. However, since the sensor signal is synthesized by a perceptual-difficult type of digital watermarking technique, the user cannot visually recognize it.

[0127] The image system 120 repeats such processing at the frame rate of the captured image. The imaging device 71 captures the robot 1 and the object W arranged at the second site S2, and the display device 8 displays the captured image at the first site S1. When the user operates the operating device 2 at or after operating at the first site S1, the user can confirm the images of the robot 1 and the object W during or after grinding at the second site S2 on the display device 8 arranged at the first site S1.

[0128] [Delay control] As described above, when the master-slave control and the image display control are performed in parallel, compared with the time from when the contact force sensor 13 acquires the sensor signal until the control device 3 executes the operation control of the robot 1 and the operating device 2 using the sensor signal, the time from when the imaging device 71 acquires the captured image until the captured image is displayed on the display device 8 takes longer.

[0129] Specifically, when the robot 1 grinds the object W, the contact force sensor 13 detects the contact force from the object W as a sensor signal, and the imaging device 71 acquires a captured image of the robot 1 and the object W as an image signal. The sensor signal of the contact force sensor 13 is input to the control device 3 via the first transmission path of the master-slave system 110. The control device 3 controls the robot 1 and the operating device 2 with the sensor signal as one of the inputs as described above. On the other hand, the image signal of the imaging device 71 is input to the display device 8 via the second transmission path of the image system 120. Further, the image signal is encoded by the image processing device 72 and decoded by the display control device 82 in the middle thereof.

[0130] Here, compared with the sensor signal, the data amount of the image signal is considerably large. In particular, when the captured image has high image quality, the difference in the data amount becomes even larger. Therefore, the communication time of the image signal from the image processing device 72 to the display control device 82 becomes longer than the communication time of the sensor signal from the robot control device 14 to the control device 3. In addition, the image signal needs to be encoded and decoded. When the data amount of the image signal is large, the time required for encoding and decoding also becomes longer.

[0131] As a result, with respect to the sensor signal and the captured image that were originally acquired at approximately the same timing, the display timing of the captured image by the display device 8 is shifted, specifically, delayed, with respect to the master-slave control using the sensor signal. FIG. 11(i) is an example of a sensor signal acquired by the control device 3 from the robot control device 14. FIG. 11(ii) is an example of the image signal after decoding by the display control device 82. As shown in FIG. 11(i), the sensor signal acquired by the control device 3 is slightly delayed compared to the sensor signal of the contact force sensor 13 output from the input processing unit 41 shown in FIG. 9(i) due to the influence of communication time between the robot control device 14 and the control device 3 and the like. However, the delay is small enough to be ignored. On the other hand, as shown in FIG. 11(ii), the decoded image signal is delayed compared to the image signal output from the imaging device 71 shown in FIG. 9(ii) due to the influence of encoding time, communication time between the image processing device 72 and the display control device 82, and decoding time. The delay of the image signal is larger compared to the delay of the sensor signal.

[0132] Therefore, the user will perform grinding using the robot 1 and the operation device 2 while observing the captured image of the robot 1 and the object W that is delayed compared to real time. Furthermore, the user will obtain two pieces of feedback information, namely, image display by the display device 8 and reaction force presentation via the operation device 2, with respect to the operation of the operation device 2. However, since the timing between the reaction force presentation and the image display is shifted, the user feels a sense of discomfort. In addition, humans tend to rely more on visual information than on tactile information. Therefore, the user will operate the operation device 2 relying on the image of the display device 8 that is delayed from real time rather than the reaction force presented by the operation device 2. The higher the image quality, the greater this tendency becomes. However, as the image quality becomes higher, as described above, the delay of the image display by the display device 8 becomes even larger.

[0133] Therefore, in the remote control system 100, the control device 3 delays the master-slave control so as to reduce the deviation in the display timing of the captured image by the display device 8 with respect to the master-slave control. As an example, in the step Sa3 of the master-slave control described above, the delay unit 69 delays the input of the combined force fm + fs from the addition unit 63 to the force / velocity conversion unit 64. In this way, the control device 3 includes a delay element such as a low-pass filter of the delay unit 69 between the input of the operating force and the contact force and the operation of the robot 1 and the operating device 2. As a result, the control device 3 delays the operation control of the robot 1 and the operating device 2, that is, delays the time from receiving the sensor signal from the contact force sensor 13 to operating the robot 1 and the operating device 2.

[0134] This means that the responsiveness of the master-slave control decreases. However, the responsiveness of the image display control (that is, the responsiveness of the image display by the display device 8 with respect to the image capture by the imaging device 71) and the responsiveness of the master-slave control are made to match or be close. Even when the user operates the operating device 2 relying on the captured image displayed on the display device 8, the sense of discomfort given to the user can be reduced. For example, it is possible to prevent the object W from being ground more than what is displayed on the display device 8.

[0135] In particular, when the captured image has high image quality, the time required for encoding the image signal, communicating the image signal, and decoding the image signal becomes longer. Even when communication time is reduced due to the development of communication technology, the time required for encoding and decoding the image signal remains long. In addition, due to the higher image quality of the captured image, the user's dependence on the captured image when operating the operating device 2 becomes higher. Therefore, the user is more likely to perceive a deviation in the display timing of the captured image by the display device 8 with respect to the master-slave control. On the other hand, the remote control system 100 reduces the deviation in the display timing of the captured image by the display device 8 with respect to the master-slave control while allowing a decrease in the responsiveness of the master-slave control by delaying the master-slave control. As a result, it is possible to reduce the discomfort when the user operates the operating device 2 while viewing the captured image on the display device 8.

[0136] [Derivation of Delay Amount] In addition, while performing such master-slave control and image display control, the control device 3 obtains the delay amount of the master-slave control in the delay control.

[0137] Specifically, explaining according to the flowchart of FIG. 6, in step Sa1, the input processing unit 41 transmits the sensor signal of the contact force sensor 13 to the control device 3 via the communication unit 19 and inputs it to the image processing device 72. In the image processing device 72, as described above, in step Sb2, the combining unit 77 associates the captured image of the imaging device 71 and the detection result of the contact force sensor 13 acquired at corresponding timings. Specifically, the image processing device 72 adds the sensor signal of the contact force sensor 13 acquired at the timing corresponding to the captured image to the captured image. More specifically, the combining unit 77 synthesizes the sensor signal acquired at substantially the same timing as each frame of the image signal with the image of each frame of the image signal as an image by an electronic watermark technique.

[0138] In the image system 120, the image signal synthesized from the sensor signals is encoded (step Sb3), wirelessly communicated (step Sb4), and decoded (step Sb5). After that, the decoded image signal is input not only from the display control device 82 to the display 81, but also from the display control device 82 to the control device 3.

[0139] In the master-slave system 110, the separation unit 67 separates the sensor signal from the decoded image signal as shown in Fig. 11(ii) as shown in Fig. 12. In this example, the sensor signal is separated from the image signal based on the electronic watermark technology. The separated sensor signal is input from the separation unit 67 to the deviation detection unit 68. The sensor signal input from the input processing unit 41 to the control device 3 is also input to the deviation detection unit 68 via the contact force acquisition unit 62. That is, the sensor signal of the contact force sensor 13 is input to the deviation detection unit 68 via two transmission paths: the first transmission path including the robot 1, wireless communication, and the control device 3, and the second transmission path including the image processing device 72, wireless communication, and the display control device 82. As described above, the sensor signal passing through the second transmission path has undergone encoding and decoding processes on the way.

[0140] Then, in step Sa5, the deviation detection unit 68 determines whether or not a predetermined detection condition is satisfied. In this example, the detection condition is that the deviation detection unit 68 receives the sensor signal from the separation unit 67.

[0141] Since the control cycle of the master-slave control is shorter than the cycle corresponding to the frame rate of the image signal, the deviation detection unit 68 receives the sensor signal from the contact force acquisition unit 62 (i.e., the sensor signal passing through the first transmission path) at a shorter cycle than the sensor signal from the separation unit 67 (i.e., the sensor signal passing through the second transmission path). Therefore, when the detection condition is not satisfied, that is, when no sensor signal is input from the separation unit 67 to the deviation detection unit 68, the deviation detection unit 68 stores the sensor signal input from the contact force acquisition unit 62 in the memory 33 and ends the derivation of the delay amount in the current control cycle.

[0142] When the detection condition is satisfied, that is, when a sensor signal is input from the separation unit 67 to the deviation detection unit 68, in step Sa6, the deviation detection unit 68 compares the sensor signal from the separation unit 67 with the sensor signal from the contact force acquisition unit 62 stored in the memory 33, and obtains the deviation time of the sensor signal from the separation unit 67 with respect to the sensor signal from the contact force acquisition unit 62.

[0143] Specifically, as shown in FIG. 12, the deviation detection unit 68 periodically receives a sensor signal from the separation unit 67, specifically, at the frame rate of the captured image. In the memory 33, a sensor signal as shown in FIG. 11(i) is stored. For example, the deviation detection unit 68 searches for a signal waveform that matches the signal waveform of the sensor signal from the separation unit 67 among the sensor signals from the contact force acquisition unit 62 stored in the memory 33. Then, the deviation detection unit 68 obtains the time difference between the sensor signal from the separation unit 67 and the portion where the signal waveforms of the sensor signals from the contact force acquisition unit 62 match as the deviation time. The deviation detection unit 68 stores the obtained deviation time in the storage unit 32.

[0144] The deviation detection unit 68 repeats such derivation of the deviation time each time the detection condition is satisfied. As a result, the deviation time accumulates in the storage unit 32.

[0145] In step Sa7, the deviation detection unit 68 determines whether a predetermined update condition is satisfied. For example, the update condition is that a state in which the deviation time changes beyond a predetermined variation range (hereinafter referred to as "variation threshold") with respect to the current delay time continues for a predetermined period (hereinafter referred to as "period threshold") or more. In short, the update condition is that a state in which the deviation time is significantly different from the current delay time continues for a certain period or more.

[0146] Specifically, when the deviation detection unit 68 obtains the deviation time, it compares the deviation time with the current delay time, determines whether the time difference exceeds the fluctuation threshold value, and stores the determination result in the storage unit 32. Further, when the time difference exceeds the fluctuation threshold value, the deviation detection unit 68 reads out the determination results during the period from the current time back by the period threshold value from the storage unit 32, and determines whether the state where the time difference exceeds the fluctuation threshold value has continued for the period threshold value or more. Note that when the positive or negative of the time difference has changed during the period threshold value, the state where the time difference exceeds the fluctuation threshold value is not treated as continuing.

[0147] When the state where the time difference exceeds the fluctuation threshold value has not continued for the period threshold value or more, the deviation detection unit 68 ends the derivation of the delay amount in the current control cycle.

[0148] When the state where the time difference exceeds the fluctuation threshold value has continued for the period threshold value or more, the deviation detection unit 68 averages the deviation times during the period threshold value in step Sa8, and updates the current delay time with the average value. In this example, the delay time corresponds to the delay amount of the master-slave control in the delay control.

[0149] The deviation detection unit 68 obtains the time constant from the updated delay time using the conversion function stored in the storage unit 32. The deviation detection unit 68 updates the time constant of the delay unit 69 with the obtained time constant.

[0150] At this time, the deviation detection unit 68 sets a limit on the change width of the delay time. That is, the deviation detection unit 68 gradually changes the current delay time to the new delay time over a plurality of control cycles so that the responsiveness of the master-slave control does not change abruptly.

[0151] In this way, while performing master-slave control and image display control, the remote control system 100 obtains the delay amount of the master-slave control by using the signals of the master-slave control and the image display control. Specifically, the deviation detection unit 68 of the control device 3 obtains the deviation time between the sensor signal received by the control device 3 from the contact force sensor 13 for master-slave control and the image signal transmitted from the imaging device 71 to the display device 8 and displayed for image display control, and obtains the delay amount based on the obtained deviation time. During the execution of the master-slave control and the image display control, since the delay amount is obtained based on the signals transmitted and received and processed for the master-slave control and the image display control, the delay amount conforming to the actual control situation can be obtained. For example, the communication environment between the robot control device 14 and the control device 3, and the communication environment between the image processing device 72 and the display control device 82 can change. Even when these communication environments change, by obtaining the delay amount by the above-described method, the delay amount can be changed corresponding to the change in the communication environment.

[0152] Furthermore, the time required for encoding and decoding can change depending on the image captured by the imaging device 71. For example, the time required for encoding and decoding is different between an image with a uniform color tone and an image with various colors and brightness levels. In grinding, sparks can occur at the contact portion between the object W and the grinding device 11a. That is, the time required for encoding and decoding is different depending on the presence or absence of sparks or the amount of sparks included in the image. Even in such a case, by obtaining the delay amount by the above-described method, the delay amount can be changed corresponding to the situation of the image.

[0153] Also, by associating the sensor signal and the image signal acquired at corresponding timings by the contact force sensor 13 and the imaging device 71 with each other, the control device 3 can discriminate the sensor signal and the image signal associated with each other from the sensor signal used for master-slave control and the image signal displayed on the display device 8, and obtain their deviation time.

[0154] Specifically, among the sensor signal and the image signal acquired at corresponding timings by the contact force sensor 13 and the imaging device 71, the sensor signal is added to the image signal as the association information indicating the association therebetween. The control device 3 obtains the deviation time between the sensor signal and the image signal based on the sensor signal as the association information. That is, when receiving both the sensor signal and the image signal, the control device 3 can discriminate the image signal originally acquired at the timing corresponding to the sensor signal by comparing the sensor signal added to the sensor signal and the image signal, and can obtain the deviation time between the two. In this example, the control device 3 separates the sensor signal from the image signal by the separation unit 67, and obtains the deviation time between the sensor signal from the contact force acquisition unit 62 and the sensor signal separated from the image signal as the deviation time between the sensor signal and the image signal.

[0155] At this time, by using the sensor signal as the association information, there is no need to add other information to the sensor signal, and it is only necessary to add the sensor signal to the image signal. Further, there is no need to prepare other information other than the sensor signal and the image signal. Therefore, the processing is simplified.

[0156] Furthermore, when a plurality of remote control systems 100 are installed, there is a risk of contamination in wireless communication. Even in such a case, by using the sensor signal as the association information, it is possible to easily discriminate whether the combination of the sensor signal and the image signal is appropriate (that is, whether contamination has occurred). For example, when the acquisition time is adopted as the association information and acquisition time information is added to each of the sensor signal and the image signal, it is necessary to further add identification information for identifying each of the remote control systems 100 to the sensor signal and the image signal. When the sensor signal is used as the association information, the sensor signal functions as the identification information of the remote control system 100. Therefore, additional identification information for identifying each of the remote control systems 100 becomes unnecessary.

[0157] In the remote control system 100, the addition of the sensor signal to the image signal is performed before the encoding of the image signal, and the separation of the sensor signal from the image signal is performed after the decoding of the image signal. Therefore, the sensor signal separated by the separation unit 67 is delayed by the time required for the encoding of the image signal, the wireless communication of the image signal, and the decoding of the image signal. Therefore, the control device 3 can obtain a deviation time that appropriately reflects the influence of the encoding, wireless communication, and decoding of the image signal.

[0158] Furthermore, the addition of the sensor signal to the image signal is realized by an electronic watermark technique. Therefore, even when the display device 8 displays the image signal in which the sensor signal is synthesized on the display 81, the sensor signal does not interfere with the presentation of the captured image to the user. That is, in the display device 8, special processing such as separating the sensor signal from the image signal becomes unnecessary.

[0159] In addition, when the state in which the deviation time changes beyond a predetermined variation threshold with respect to the current delay time continues for a period threshold or more, the control device 3 updates the delay amount of the master-slave control, that is, the delay time. Thereby, small variations in the deviation time, which are allowed in normal master-slave control and image display control, are ignored, the delay amount of the master-slave control is kept constant, and the master-slave control can be stabilized. That is, the communication environment between the image processing device 72 and the display control device 82 is not constant and can have some variations. Also, the time required for encoding and decoding the image signal can vary depending on the captured image. Therefore, when the variation amount of the deviation time is small and the variation of the deviation time is temporary, the control device 3 can keep the responsiveness of the master-slave control constant by keeping the delay amount of the master-slave control constant, and prevent giving the user a sense of discomfort.

[0160] Furthermore, when updating the delay time, the control device 3 can reduce the discomfort given to the user by changing the delay time step by step. That is, the delay time is directly related to the responsiveness of the master-slave control. By changing the delay time step by step, the control device 3 can prevent the responsiveness of the master-slave control from changing suddenly.

[0161] As described above, the remote control system 100 includes an operating device 2 (master device) operated by a user, a robot 1 (slave device) that applies an action to the object W according to the operation of the operating device 2, a contact force sensor 13 (sensor) provided in the robot 1 for detecting the operating state of the robot 1, an imaging device 71 for capturing at least one of an image of the robot 1 and the object W, a display device 8 for displaying the captured image by the imaging device 71 and providing it to the user who operates the operating device 2, and a control device 3 for executing at least one of the operation controls of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13. The control device 3 delays the operation control so as to reduce the deviation in the display timing of the captured image by the display device 8 for the operation control.

[0162] In other words, the control method of the remote control system 100 including the operating device 2 operated by the user, the robot 1 that applies an action to the object W according to the operation of the operating device 2, the contact force sensor 13 provided in the robot 1 for detecting the operating state of the robot 1, the imaging device 71 for capturing at least one of an image of the robot 1 and the object W, and the display device 8 for displaying the captured image by the imaging device 71 and providing it to the user who operates the operating device 2 includes executing at least one of the operation controls of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13, and delaying the operation control so as to reduce the deviation in the display timing of the captured image by the display device 8 for the operation control.

[0163] Alternatively, a remote control program for causing a computer to realize a function of controlling a remote control system 100 including an operation device 2 operated by a user, a robot 1 that applies an action to an object W according to the operation of the operation device 2, a contact force sensor 13 provided in the robot 1 for detecting an operating state of the robot 1, an imaging device 71 for capturing an image of at least one of the robot 1 and the object W, and a display device 8 for displaying a captured image by the imaging device 71 and providing it to the user who operates the operation device 2 causes the computer to realize a function of performing operation control of at least one of the robot 1 and the operation device 2 based on the detection result of the contact force sensor 13, and a function of delaying the operation control so as to reduce a deviation in the display timing of the captured image by the display device 8 with respect to the operation control.

[0164] According to these configurations, when the user operates the operation device 2, the robot 1 operates to apply an action (for example, grinding) to the object W. At this time, the contact force sensor 13 detects the operating state of the robot 1 (for example, the reaction force acting on the robot 1). Based on the detection result of the contact force sensor 13, operation control of at least one of the robot 1 and the operation device 2 is executed. In parallel with this, the imaging device 71 captures an image of at least one of the robot 1 and the object W, and the display device 8 displays the captured image and provides the captured image to the user. Here, the data amount of the captured image is larger than the detection result of the contact force sensor 13. Therefore, the time required for processing and communication from when the captured image is acquired by the imaging device 71 until it is displayed on the display device 8 is longer than the time required for processing and communication from when the detection result is acquired by the contact force sensor 13 until it is used for operation control. That is, the display timing by the display device 8 of the captured image tends to deviate (specifically, be delayed) with respect to the operation control of at least one of the robot 1 and the operation device 2 based on the detection result of the contact force sensor 13. On the other hand, the operation control is delayed so as to reduce the deviation in the display timing by the display device 8 of the captured image with respect to the operation control. As a result, although the responsiveness of the operation control decreases, it is possible to reduce the discomfort given to the user who operates the operation device 2 while viewing the captured image on the display device 8.

[0165] Further, when the control device 3 receives the detection result of the contact force sensor 13 from the robot 1 and receives the captured image from the display device 8, the control device 3 obtains a delay amount for delaying the operation control based on the deviation time between the detection result of the contact force sensor 13 and the captured image.

[0166] According to this configuration, the control device 3 receives the detection result of the contact force sensor 13 from the robot 1 and executes operation control based on this detection result. On the other hand, the control device 3 has received the captured image from the display device 8, and this captured image is the one displayed on the display device 8. That is, the deviation time between the detection result of the contact force sensor 13 received by the control device 3 and the captured image generally matches the deviation time between the operation control and the display by the display device 8 of the captured image. Therefore, the control device 3 can obtain the delay amount of the operation control corresponding to the deviation between the operation control and the display by the display device 8 of the captured image by obtaining the delay amount of the operation control based on the deviation time between the detection result of the contact force sensor 13 and the captured image.

[0167] Furthermore, at least one of the detection result and the captured image obtained at corresponding timings by the contact force sensor 13 and the imaging device 71 has association information indicating the association between them added thereto, and the control device 3 determines the deviation time by discriminating the detection result and the captured image based on the association information.

[0168] According to this configuration, the control device 3 can easily discriminate the detection result and the captured image obtained at corresponding timings from among the detection result and the captured image by referring to the association information.

[0169] Furthermore, the remote control system 100 further includes an image processing device 72 that receives a captured image from the imaging device 71, processes the captured image, and transmits the processed captured image to the display device 8. The image processing device 72 receives the detection result from the robot 1, adds the detection result obtained at the timing corresponding to the captured image as related information to the captured image, transmits the captured image with the detection result added to the display device 8, and the control device 3 receives the captured image with the detection result added from the display device 8 and obtains a deviation time based on a comparison between the detection result added to the captured image and the detection result received from the robot 1.

[0170] According to this configuration, the detection result of the contact force sensor 13 as related information indicating the association between the detection result of the contact force sensor 13 and the captured image of the imaging device 71, which are obtained at corresponding timings, is added to the captured image. Therefore, the detection result is added to the captured image transmitted to the display device 8 and received by the control device 3 from the display device 8. The control device 3 receives the detection result from the robot 1. The deviation time between the detection result received from the robot 1 and the captured image with the detection result added, that is, the deviation time between the detection result and the captured image. Therefore, the control device 3 can easily obtain the deviation time between the detection result and the captured image by comparing the detection result added to the captured image and the detection result received from the robot 1.

[0171] Also, the image processing device 72 adds the detection result to the captured image by synthesizing the detection result obtained at the timing corresponding to the captured image as an image to the captured image, encodes the captured image with the detection result synthesized, and transmits it to the display device 8, and the control device 3 obtains the detection result from the captured image decoded by the display device 8.

[0172] According to this configuration, the detection result of the contact force sensor 13 synthesized in the captured image received by the control device 3 from the display device 8 has passed through encoding, communication between the image processing device 72 and the display device 8, and decoding together with the captured image. That is, the control device 3 can obtain a deviation time reflecting the delay due to these processes and communications by obtaining the deviation time based on the comparison between the detection result added to the decoded captured image and the detection result received from the robot 1.

[0173] As operation control, the control device 3 controls the operation of the robot 1 according to the operation of the operation device 2 by the user and the detection result, and controls the operation of the operation device 2 so as to present the reaction force acting on the robot 1 to the user.

[0174] According to this configuration, the operation control by the control device 3 includes control of the operation of the robot 1 and control of the operation of the operation device 2. That is, when a deviation occurs in the display timing of the captured image by the display device 8 with respect to the operation control, the control of the operation of the robot 1 and the display timing of the captured image by the display device 8 deviate, and the control of the operation of the operation device 2 and the display timing of the captured image by the display device 8 deviate. For example, when the display timing of the captured image by the display device 8 is delayed with respect to the control of the operation of the robot 1, the user will operate the operation device 2 while observing the operation of the robot 1 or the captured image of the object W that is later than the actual operation of the robot 1. Also, when the display timing of the captured image by the display device 8 is delayed with respect to the control of the operation of the operation device 2, the user senses the reaction force from the operation device 2 while observing the operation of the robot 1 or the captured image of the object W that is later than the operation of the robot 1 when receiving the reaction force presented from the operation device 2. In either case, the user will feel a sense of discomfort. On the other hand, since the control device 3 delays the operation control, the control of the operation of the robot 1 and the control of the operation of the operation device 2 will both be delayed. As a result, when the user operates the operation device 2 while observing the captured image of the display device 8, the deviation between the operation of the robot 1 and the captured image, and the deviation between the reaction force felt from the operation device 2 and the captured image can be reduced.

[0175] The contact force sensor 13 is an example of a sensor that detects the operating state of the robot 1, and further is an example of a force sensor.

[0176] <<Other Embodiments>> As described above, the above-described embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and is also applicable to embodiments in which appropriate changes, replacements, additions, omissions, etc. are made. Further, it is also possible to combine the respective components described in the above-described embodiments to form a new embodiment. Further, among the components described in the accompanying drawings and the detailed description, there may be included not only the components essential for solving the problems, but also the components not essential for solving the problems for exemplifying the technology. Therefore, it should not be immediately determined that those non-essential components are essential just because they are described in the accompanying drawings and the detailed description.

[0177] The configuration of the remote control system 100 described above is merely an example and is not limited thereto. For example, regarding the master-slave system 110, the action applied by the robot 1 to the object W is not limited to grinding. The action applied by the robot 1 to the object W may be cutting or polishing, or may be pressing or gripping.

[0178] Further, the communication between the robot 1 and the control device 3 is not limited to wireless communication, and may be wired communication. Similarly, the communication between the imaging device 71 and the display device 8 is not limited to wireless communication, and may be wired communication.

[0179] The operation control of the robot 1 and the operation device 2 executed by the control device 3 is not limited to bilateral control. For example, the control device 3 may independently perform the operation control of the robot 1 according to the operation of the operation device 2 and the operation control of the operation device 2 according to the reaction force received by the robot 1 from the object W.

[0180] Alternatively, the control device 3 may only perform the operation control of the robot 1 according to the operation of the operation device 2, without presenting the reaction force to the user via the operation device 2. That is, the control device 3 may only perform the operation control of the robot 1 based on the operation of the operation device 2 by the user and the sensor signal of the contact force sensor 13.

[0181] Further, the robot 1 is not limited to an industrial robot. For example, the robot 1 may be a medical robot. Furthermore, the slave device is not limited to a robot. For example, the slave device may be a machine such as a construction machine.

[0182] The sensor for detecting the operating state of the slave device is not limited to the contact force sensor 13, and any sensor may be adopted as long as it can detect the operating state of the slave device. For example, when the slave device is the robot 1, the sensor for detecting the operating state of the slave device may be a current sensor for detecting the current of the servo motor 15 or a torque sensor for detecting the torque of the servo motor 15.

[0183] Furthermore, the sensor signal synthesized into the captured image is not limited to the sensor signal of the force in the Z-axis direction. The sensor signal synthesized into the captured image may be a sensor signal of the force in a direction other than the Z-axis direction, or may include sensor signals of the force in a plurality of directions not limited to the Z-axis direction. For example, the user may select the sensor signal synthesized into the captured image from among the six-axis force sensor signals in step Sa0 or the like. Alternatively, the robot control device 14 or the image processing device 72 may select the sensor signal synthesized into the captured image from among the six-axis force sensor signals. For example, the robot control device 14 or the image processing device 72 may select the sensor signal with a large variation from among the six-axis force sensor signals of the contact force sensor 13.

[0184] The synthesis of the sensor signal into the captured image is not limited to that by the perceptual-difficult type of electronic watermarking technology. For example, the sensor signal may be synthesized into the captured image in a visible state. The synthesis of the sensor signal in a visible state may be realized by the perceptual type of electronic watermarking technology or may be realized by other methods.

[0185] In the above example, the sensor signal added to each frame of the image signal includes a plurality of numerical data and forms a distinguishable signal waveform from the sensor signals added to other frames. However, depending on the sampling period of the sensor signal and the frame rate of the image signal, the sensor signal added to each frame of the image signal may have only a small amount of numerical data and may not be able to form a distinguishable signal waveform from the sensor signals added to other frames. In such a case, the deviation detection unit 68 may combine the sensor signal added to the frame for which the deviation time is to be derived with the sensor signals added to one or more immediate previous frames to form a distinguishable signal waveform, and then compare the combined sensor signal with the sensor signal from the contact force acquisition unit 62.

[0186] The correlation information indicating the association between the detection result of the sensor and the captured image of the imaging device, which are obtained at corresponding timings, is not limited to the detection result of the sensor (the sensor signal of the contact force sensor 13 in the above example). For example, the correlation information may be time information. That is, time information at which the detection result is obtained may be added to the detection result of the sensor, and time information at which the captured image is obtained may be added to the captured image of the imaging device. In that case, it is not necessary to add the detection result of the sensor to the image signal. For example, the robot control device 14 may add time information at which the sensor signal is obtained to the sensor signal of the contact force sensor 13, and the image processing device 72 may add time information at which the image signal is obtained to the image signal of the imaging device 71. Input of the sensor signal from the robot control device 14 to the image processing device 72 becomes unnecessary. The deviation detection unit 68 can determine the sensor signal and the image signal obtained at corresponding timings by comparing the time signal added to the sensor signal with the time signal added to the image signal, and obtain the deviation time between the two. The addition of time information to the image signal may be adding the time information as data, or synthesizing the time information as an image with the captured image. Note that the correlation information may be identification information indicating the acquisition timing other than the time information.

[0187] The update condition of the delay time is not limited to a state where the deviation time changes beyond a predetermined variation range with respect to the current delay time and continues for a predetermined period or more. For example, the update condition may be that the deviation time changes beyond a predetermined variation range with respect to the current delay time. Alternatively, the update condition may be the arrival of a predetermined cycle regardless of the variation of the deviation time. That is, the delay time may be updated every predetermined cycle.

[0188] When updating the delay time on the condition that a state where the deviation time changes beyond a predetermined variation range with respect to the current delay time continues for a predetermined period or more, the new delay time does not have to be the average value obtained by averaging the deviation times during the predetermined period. For example, the new delay time may be the latest deviation time, or the deviation time with the highest frequency during the predetermined period.

[0189] Furthermore, when updating the delay time, instead of updating the current delay time step by step to the new delay time, it may be updated all at once.

[0190] Also, the update of the delay time may not be automatically performed by the control device 3. Instead, when the update condition is satisfied, a new delay time may be presented to the user, and the delay time may be updated by an operation from the user.

[0191] The above-described block diagram is an example, and a plurality of blocks may be realized as one block, one block may be divided into a plurality of blocks, or some functions may be transferred to another block.

[0192] The technology of the present disclosure may be a program for executing the control method, or may be a non-temporary computer-readable recording medium on which the program is recorded. Further, the program may be distributed via a transmission medium such as the Internet.

[0193] The functions of the configuration disclosed in the present embodiment may be executed using an electric circuit or a processing circuit. The processor is a processing circuit including transistors and other circuits. In the present disclosure, a unit, a controller, or a means is hardware or a programmed one for executing the described functions. Here, the hardware is the one disclosed in the present embodiment or known hardware configured or programmed to execute the functions disclosed in the present embodiment. When the hardware is a processor or a controller, the circuit, the means, or the unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

Explanation of Reference Numerals

[0194] 100 Remote control system 1 Robot (slave device) 13 Contact force sensor (sensor) 2 Operating device (master device) 3 Control device 71 Imaging device 72 Image processing device 8 Display device W Object

Claims

1. A master device operated by a user, A slave device that applies an action to an object according to the operation of the master device, A sensor provided in the slave device for detecting the operating state of the slave device, An imaging device that captures an image of at least one of the slave device and the object, A display device that displays the captured image by the imaging device and provides it to the user who operates the master device, A control device that executes operation control of both the master device and the slave device based on the detection result of the sensor, The control device is a remote control system that delays the operation control so as to reduce a deviation in the display timing of the captured image by the display device for the operation control.

2. In the remote control system according to Claim 1, The control device receives the detection result from the slave device and receives the captured image from the display device, and obtains a delay amount for delaying the operation control based on a deviation time between the detection result and the captured image. A remote control system.

3. In the remote control system according to Claim 2, When the control device updates the delay amount with a new delay amount obtained based on the deviation time, the control device gradually updates the delay amount to the new delay amount. Remote control system.

4. In the remote control system according to Claim 2, At least one of the detection result and the captured image acquired at corresponding timings by the sensor and the imaging device is added with association information indicating the association with each other, The control device discriminates the detection result and the captured image acquired at corresponding timings from among the received detection result and the captured image based on the association information, and obtains the deviation time. Remote control system.

5. In the remote control system according to Claim 4, The remote control system further includes an image processing device that receives the captured image from the imaging device, processes the captured image, and transmits the processed captured image to the display device, The image processing device receives the detection result from the slave device, adds the detection result acquired at the timing corresponding to the captured image as the association information to the captured image, and transmits the captured image with the detection result added thereto to the display device. The control device receives the captured image with the detection result added from the display device, and obtains the deviation time based on a comparison between the detection result added to the captured image and the detection result received from the slave device. A remote control system.

6. In the remote control system according to claim 5, The image processing device adds the detection result to the captured image by synthesizing the detection result acquired at the timing corresponding to the captured image as an image to the captured image, encodes the captured image with the detection result synthesized therein, and transmits it to the display device. The control device obtains the detection result from the captured image decoded by the display device. A remote control system.

7. In the remote control system according to any one of claims 1 to 6, As the operation control, the control device controls the operation of the slave device according to the operation of the master device by the user and the detection result, and controls the operation of the master device so as to present the reaction force acting on the slave device to the user. A remote control system.

8. In the remote control system according to claim 7, The sensor is a force sensor. A remote control system.

9. A master device operated by a user, A slave device that applies an action to an object according to the operation of the master device, A sensor provided in the slave device for detecting the operating state of the slave device, An imaging device that captures an image of at least one of the slave device and the object, A control method for a remote control system including a display device that displays a captured image by the imaging device and provides it to a user who operates the master device, Performing operation control of both the master device and the slave device based on the detection result of the sensor; Delaying the operation control so as to reduce a deviation in display timing of the captured image by the display device with respect to the operation control. A control method for a remote control system.

10. A master device operated by a user, A slave device that applies an action to an object according to the operation of the master device, A sensor provided in the slave device for detecting the operating state of the slave device, An imaging device that captures an image of at least one of the slave device and the object, A remote control program for causing a computer to realize a function of controlling a remote control system including a display device that displays a captured image by the imaging device and provides it to a user who operates the master device. A function of executing operation control of both the master device and the slave device based on the detection result of the sensor, A remote control program for causing a computer to realize a function of delaying the operation control so as to reduce a deviation in display timing of the captured image by the display device with respect to the operation control.

Images