Remote operation system and remote operation method

Abstract

To provide a remote operation system capable of notifying a user of interference or collision between a robot and a surrounding environment or a work object when the user manually operates a robot in a remote location.SOLUTION: A remote operation system includes: a prediction simulation unit that predicts interference between a model of a robot in a simulation space and a model of a surrounding environment on the basis of state information received from built-in sensors of the robot and motion commands received from an operation interface; and a robot control unit that controls the robot on the basis of the motion commands received from the operation interface in a normal operation mode and controls the robot on the basis of adjusted motion commands received from an operation amount adjustment unit in a work operation mode. The prediction simulation unit inquires of the user whether the predicted interference is interference intended by the user via a display device and causes a transition from the normal operation mode to the work operation mode when it is determined that the interference is intended interference.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a remote control system and a remote control method that can avoid interference between the robot and the surrounding environment or a work object when a user manually operates a remotely located robot. [Background technology]

[0002] It has been common for a user to manually operate a robot to perform a desired task, but manual operation of a robot can sometimes cause unintended interference or collision between the robot and the surrounding environment or the work target, creating a need for robot control technology to avoid unintended interference and collision.

[0003] As an example of this technology, the abstract of Patent Document 1 states, "A robot system includes a user interface configured to receive jog commands for manually operating a robot unit, and a control unit coupled to the user interface. The control unit performs real-time parallel processing of the jog commands, which includes executing a collision check thread to determine, based on an environmental model and a robot model, whether the jog command will result in a collision or an obstacle-free status for the robot unit in the operating environment, and executing a jog operation thread to determine whether the obstacle-free status is provided within a collision check time limit. The control unit also executes a jog command by the robot unit based on the obstacle-free status provided before the collision check time limit." Thus, a robot system is disclosed that has a function for determining, based on an environmental model and a robot model, whether a collision will occur when a command is executed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-169146 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Figure 1 of Patent Document 1, the robot unit 110 and the controller 112 are positioned close to each other, and therefore, it is thought that the robot system in this document is intended to be operated manually by a user in the vicinity of the robot while visually observing the robot's behavior. Therefore, in this document, there was no need to notify the user who could visually observe the robot's behavior of the collision determination result by the robot system.

[0006] In recent years, however, there has been an increase in situations where robots are manually operated in remote locations, and even in such situations, there is a demand for robot control technology that can avoid unintended interference and collisions.

[0007] Therefore, an object of the present invention is to provide a remote control system and a remote control method that, when a user manually operates a robot at a remote location, can notify the user of interference or collision between the robot and the surrounding environment or a work object, and can switch subsequent robot control depending on whether the interference or collision was intended by the user. [Means for solving the problem]

[0008] The present invention provides a remote operation system that solves the above problems as follows: That is, a system that includes a robot, a robot control device, an operation interface, and a display device, and is capable of controlling the robot in a normal operation mode or a work operation mode, wherein the robot control device includes a prediction simulation unit that predicts interference between a model of the robot and a model of the surrounding environment in a simulation space based on status information received from an internal sensor of the robot and an operation command received from the operation interface, a manipulation amount adjustment unit that generates an adjusted operation command from the operation command received from the operation interface, and a robot control unit that controls the robot based on the operation command received from the operation interface in the normal operation mode and the adjusted operation command received from the manipulation amount adjustment unit in the work operation mode, wherein the prediction simulation unit asks the user via the display device whether the predicted interference is intended by the user, and if it is determined that the predicted interference is intended, transitions the system from the normal operation mode to the work operation mode, and if it is determined that the predicted interference is not intended, continues the normal operation mode. [Effects of the Invention]

[0009] According to the remote control system and remote control method of the present invention, when a user manually operates a robot in a remote location, the user can be notified of interference or collision between the robot and the surrounding environment or the work object, and subsequent robot control can be switched depending on whether the interference or collision was intended by the user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic configuration diagram of a remote control system according to a first embodiment. [Figure 2] 4 is a flowchart of a normal operation mode in the first embodiment. [Figure 3A] 3 shows an example of a robot simulator area in the first embodiment. [Figure 3B] 10 is another example of the robot simulator area in the first embodiment. [Figure 4] 10 is an example of a mode transition confirmation area according to the first embodiment. [Figure 5] 10 shows an example of a mode transition notification area according to the first embodiment. [Figure 6] 4 is a flowchart of a work operation mode in the first embodiment. [Figure 7] 4 is a flowchart of an overload recovery mode in the first embodiment. [Figure 8] 10 is another example of a mode transition notification area according to the first embodiment. [Figure 9] FIG. 10 is a schematic configuration diagram of a robot according to a second embodiment. [Figure 10] 10 shows an example of a robot simulator area according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be modified within the scope of the concept and spirit of the present invention.

[0012] In the configurations of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and redundant explanations may be omitted. Furthermore, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc., disclosed in the drawings, etc. [Example]

[0013] A remote control system according to a first embodiment of the present invention will be described below with reference to FIGS.

[0014] <Remote Control System Overview> 1 is a schematic diagram of a remote operation system 100 of this embodiment. As shown here, the remote operation system 100 of this embodiment is a system including a robot control device 1, an operation interface 2, a display device 3, and a robot 4. In the following, the explanation will be given assuming that the operation interface 2 and the display device 3 are installed near a user U, and that the robot 4 and a work target object 5 are installed in a remote location that is out of the user U's sight.

[0015] The robot control device 1 is a device equipped with a predictive simulation unit 11, a robot control unit 12, and an operation amount adjustment unit 13, and controls the robot 4 by automatically or manually selecting one of the robot control modes: a normal operation mode M1, a work operation mode M2, or an overload recovery mode M3. Specifically, the robot control device 1 is a computer equipped with hardware such as a calculation device such as a CPU, a storage device such as a semiconductor memory, and a communication device. The calculation device executes a predetermined program to realize each functional unit, such as the predictive simulation unit 11, but the following description will omit such well-known techniques as appropriate.

[0016] The operation interface 2 is a device for inputting operation commands when the user U manually operates the robot 4, and specifically, is a remote control, a reader arm, or a GUI operation screen.

[0017] The display device 3 is a display that notifies the user U of desired information by displaying the output of the robot control device 1 as an image.

[0018] The robot 4 is, for example, a robot equipped with a manipulator arm, and is equipped with various built-in sensors 41 for measuring state information such as the angle and torque of each joint, and the force of the hand.

[0019] Hereinafter, robot control in a case where a user U operates the operation interface 2 while looking at the display device 3 to cause a remotely located robot 4 to perform a desired task will be described for each mode.

[0020] <Normal operation mode M1> First, the normal operation mode M1, which is the control mode at system startup, will be described using the flowchart of FIG.

[0021] <<Step S11>> First, in step S11, the robot control device 1 receives an operation command for the robot 4 input by the user U operating the operation interface 2. The operation command received here is transmitted to the prediction simulation unit 11 and the robot control unit 12. Thereafter, the processing of steps S12 to S15 by the prediction simulation unit 11 and the processing of steps S16 to S18 by the robot control unit 12 are carried out in parallel.

[0022] <<Step S12>> In step S12, the predictive simulation unit 11 predicts and simulates interference between the robot 4 and the surrounding environment based on the received operation command and the state information received from the built-in sensor 41 of the robot 4.

[0023] Here, the predictive simulation unit 11 will be described in detail. The predictive simulation unit 11 stores in advance a model of the robot 4 (hereinafter referred to as the "robot model 4m") and a model of the work target object 5 (hereinafter referred to as the "work target object model 5m"). This predictive simulation unit 11 has a function of constantly updating the current posture of the robot model 4m in the simulation space based on status information constantly received from the built-in sensor 41, and also has a function of constantly predicting whether or not there will be future interference between models in the simulation space based on operation commands constantly received from the operation interface 2.

[0024] Note that by making the sizes of the robot model 4m and work object model 5m held by the predictive simulation unit 11 slightly larger than the actual robot 4 and work object 5, interference in the simulation space can be detected earlier than the actual interference occurs. Therefore, by operating the operation interface 2 while viewing the simulation results displayed on the display device 3, the user U can safely operate the robot 4 while avoiding unintended interference.

[0025] 3A is an example of the robot simulator area 31 displayed on the display device 3. This example illustrates the state of the robot model 4m and the work target object model 5m in the simulation space when no operation command is input and the robot 4 is at rest.

[0026] FIG. 3B is another example of the robot simulator area 31 displayed on the display device 3. This example illustrates the state of the current robot model 4m (solid line), the state of the future robot model 4m' (dashed line), and the state of the work target object model 5m in the simulation space when an action command to move the hand of the robot 4 in the direction of arrow A is input. In this example, the predictive simulation unit 11 predicts interference between the future robot model 4m' and the work target object model 5m, and indicates the interfering area with a circle. Note that the interference notification method is not limited to this; for example, the user U may be notified of the interference prediction result by highlighting the color of the interfering area or by flashing the interfering area.

[0027] The future robot model 4m' is calculated by the predictive simulation unit 11 by linearly predicting the state of the current robot model 4m, assuming that the same operation command will be input continuously for several seconds.

[0028] <<Step S13>> In step S13, the prediction simulation unit 11 determines whether interference was predicted in step S12. If interference was predicted, the process proceeds to step S14, and if interference was not predicted, the process returns to step S11.

[0029] <<Step S14>> In step S14, the prediction simulation unit 11 transmits an operation command to the robot control unit 12 to immediately stop the operation of the robot 4. As a result, the operation of the robot 4 is temporarily stopped.

[0030] <<Step S15>> In step S15, the prediction simulation unit 11 determines whether the predicted interference is the one intended by the user U. Specifically, the prediction simulation unit 11 presents a mode transition confirmation area 32, as shown in FIG. 4, on the display device 3, and asks the user U whether the predicted interference is the one intended. If the "Yes" button is pressed and it is confirmed that the interference is intended, the mode transitions to the work operation mode M2; if the "No" button is pressed and it is confirmed that the interference is unintended, the process returns to step S11. In the latter case, the user U can simply issue an operation command to the stopped robot 4 to avoid the predicted interference.

[0031] For example, if the user U inputs an action command to press the hand of the robot 4 against the top surface of the work target object 5, the hand of the future robot model 4m' in the simulation space will inevitably interfere with the work target object model 5m, so when the interference is predicted, the robot 4 will stop temporarily, and if the predicted interference is confirmed to be intended, the robot control mode will automatically transition to the work operation mode M2.

[0032] <<Step S16>> Meanwhile, in step S16, the robot control unit 12 controls the operation of the robot 4 based on the received operation command, and detects the load on each part of the robot 4 based on the status information received from the built-in sensor 41 at that time.

[0033] <<Step S17>> In step S17, the robot control unit 12 determines whether the detected load on each part is an overload exceeding a predetermined threshold value set for each part. If an overload is detected on any part of the robot 4, the process proceeds to step S18, and if no overload is detected, the process returns to step S11.

[0034] <<Step S18>> In step S18, the robot control unit 12 displays a mode transition notification area 33 on the display device 3, as shown in Fig. 5, to notify the user U of the transition to overload recovery mode M3 and the direction of the overload. If the user U understands this information and presses the "yes" button, the robot control mode automatically transitions to overload recovery mode M3.

[0035] <Work operation mode M2> Next, the work operation mode M2, which is a control mode intended for use when the robot 4 and the work target object 5 are close to each other, will be described using the flowchart in Fig. 6. Note that the same explanations as for the normal operation mode M1 will be omitted as appropriate for the commonalities between them.

[0036] <<Step S21>> In step S21, the robot control device 1 receives an operation command for the robot 4 that is input by the user U operating the operation interface 2. The operation command received here is transmitted to the operation amount adjustment unit 13.

[0037] <<Step S22>> In step S22, the operation amount adjustment unit 13 generates an adjusted operation command by suppressing the operation amount and operation speed indicated by the operation command received in step S21. This adjusted operation command is sent to the prediction simulation unit 11 and the robot control unit 12.

[0038] Here, the operation amount adjustment unit 13 will be described in detail. This operation amount adjustment unit 13 is a functional unit that generates an adjusted operation command such that the operating speed of the robot 4 in the work operation mode M2 ​​is reduced to a level lower than the operating speed of the robot 4 in the normal operation mode M1. For example, if the robot operating speed according to the input operation command is too fast for the robot operating speed in the vicinity of the work target object 5, the operation amount adjustment unit 13 generates an adjusted operation command for controlling the robot 4 so that the robot operating speed does not exceed a predetermined maximum speed.

[0039] <<Steps S23 to S25, S27, S28>> Steps S23 to S25, S27, and S28 are the same as steps S12 to S14, S17, and S18 in normal operation mode M1, and detailed explanations will be omitted. Note that step S12 in normal operation mode M1 uses the operation command as is, whereas step S23 in this mode uses the adjusted operation command.

[0040] <<Step S26>> In step S26, the robot control unit 12 controls the operation of the robot 4 based on the received post-adjustment operation command, and at the same time detects the load on each part of the robot 4 based on the status information received from the built-in sensor 41.

[0041] Here, the robot control in this step is impedance control that takes into account not only the position control of the hand of the robot 4 but also the force control of the hand of the robot 4. As a result, when the robot 4 is made to perform a task such as inserting a rod-shaped part into a hole in the work object 5, the robot can automatically operate in accordance with the reaction from the work object 5.

[0042] <Overload recovery mode M3> Next, the overload recovery mode M3, which is a control mode to which the system automatically transitions when an overload is detected during execution of the normal operation mode M1 or the work operation mode M2, will be described using the flowchart in Figure 7. Note that the same explanations as for the above modes will be omitted as appropriate for points in common.

[0043] <<Steps S31, S33 to S36>> Steps S31, and S33 to S36 are the same processes as steps S21, and S23 to S26 in work operation mode M2, and therefore detailed explanations thereof will be omitted.

[0044] <<Step S32>> In step S32, the manipulated variable adjuster 13 generates an adjusted motion command including only motion commands in the direction to remove the overload by removing motion commands other than those in the direction to remove the overload. This adjusted motion command is sent to the predictive simulation unit 11 and the robot control unit 12.

[0045] <<Step S37>> In step S37, the robot control unit 12 determines whether the detected load on each part is less than a predetermined threshold value set for each part. If the overload has been removed from all parts of the robot 4, the process proceeds to step S38, and if the overload has not been removed, the process returns to step S31.

[0046] <<Step S38>> In step S38, the robot control unit 12 displays a mode transition notification area 34 on the display device 3, as shown in Fig. 8, to notify the user U that the robot will transition to the original robot control mode, the normal operation mode M1 or the work operation mode M2. If the user U understands this information and presses the "yes" button, the robot control mode will automatically transition to the normal operation mode M1 or the work operation mode M2.

[0047] According to the remote control system of this embodiment described above, when a user manually operates a robot in a remote location, the user can be notified of any interference or collision between the robot and the surrounding environment or the work object, and subsequent robot control can be switched depending on whether the interference or collision was intended by the user or not. [Example]

[0048] Second Embodiment Next, a remote control system according to a second embodiment of the present invention will be described with reference to Figures 9 and 10. Note that a duplicated description of points common to the first embodiment will be omitted.

[0049] In the first embodiment, the work target object model 5m was prepared in advance, but in the present embodiment, the work target object model 5m is not prepared in advance. That is, the remote operation system 100 of the present embodiment is a system that automatically generates a work target object model 5m corresponding to a work target object 5 that is unknown to the system, and uses the generated work target object model 5m to achieve robot control equivalent to that of the first embodiment.

[0050] 9 is a schematic diagram of the robot 4 of this embodiment. As shown in this figure, in this embodiment, a three-dimensional camera 42 is installed at the hand of the robot 4. Therefore, the remote control system 100 of this embodiment can grasp the surrounding environment of the robot 4 (for example, the work target object 5) in real time based on the image captured by the three-dimensional camera 42. Note that instead of the three-dimensional camera 42, other types of three-dimensional sensors such as LiDAR may be used.

[0051] 10 is a diagram showing a work object model 5m obtained by voxelizing the work object 5 captured by the 3D camera 42 in real time, superimposed on the simulation space in the robot simulator area 31 of the display device 3. In this embodiment, no interference detection is performed between the voxels of the work object model 5m, and the voxels are used only to detect interference with the robot 4.

[0052] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been specifically described in order to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0053] Furthermore, a part of the configuration of one embodiment can be replaced with a part of the configuration of another embodiment. Also, a part of the configuration of one embodiment can be added to a part of another embodiment. Also, a part of the configuration of each embodiment can be deleted, and a part of another configuration can be added or replaced with a part of another configuration. [Explanation of symbols]

[0054] 100 Remote Control System 1. Robot control device 11 Prediction Simulation Department 12 Robot control unit 13 Operation amount adjustment section 2 Operation Interface 3 Display device 31 Robot Simulator Area 32 Mode transition confirmation area 33, 34 Mode transition notification area 4. Robot 41 Built-in sensor 42 3D Camera 5. Work object

Claims

1. A robot, a robot control device, an operation interface, and a display device, A remote operation system capable of controlling the robot in a normal operation mode or a work operation mode, The robot control device a prediction simulation unit that predicts interference between a model of the robot and a model of the surrounding environment in a simulation space based on status information received from a built-in sensor of the robot and an operation command received from the operation interface; an operation amount adjustment unit that generates an adjusted operation command from the operation command received from the operation interface; a robot control unit that controls the robot based on an operation command received from the operation interface in the normal operation mode, and controls the robot based on an adjusted operation command received from the operation amount adjustment unit in the work operation mode, the prediction simulation unit asks the user via the display device whether the predicted interference is the user's intended interference, and if it is determined that the predicted interference is the user's intended interference, transitions from the normal operation mode to the work operation mode, and if it is determined that the predicted interference is not the user's intended interference, continues the normal operation mode.

2. 2. The remote control system according to claim 1, The adjusted operation command is an operation command in which the operation amount and operation speed indicated by the operation command received from the operation interface are suppressed.

3. 2. The remote control system according to claim 1, The predictive simulation unit predicts in real time the interference between the robot model and a model of the surrounding environment in the simulation space by linearly predicting the state of the robot model in the simulation space when operation based on the motion command is continued for a predetermined period of time.

4. 4. The remote control system according to claim 3, the robot is equipped with a three-dimensional sensor; The remote control system is characterized in that the predictive simulation unit generates a model of the surrounding environment based on the output of the three-dimensional sensor.

5. 2. The remote control system according to claim 1, The robot is equipped with built-in sensors that measure the load of each part, A remote control system characterized in that the robot control unit transitions from a normal operation mode or a work operation mode to an overload recovery mode when it detects an overload based on the output of the built-in sensor.

6. 6. The remote control system according to claim 5, A remote operation system characterized in that the robot control unit controls the robot based on an adjusted operation command that limits the operation direction indicated by the operation command received from the operation interface to the direction of removing an overload.

7. 6. The remote control system according to claim 5, A remote operation system characterized in that the robot control unit transitions to the normal operation mode or the work operation mode when it detects that an overload has been removed based on the output of the built-in sensor.

8. A remote operation method carried out by a remote operation system that includes a robot, an operation interface, and a display device and is capable of controlling the robot in a normal operation mode or a work operation mode, comprising: In the normal operation mode, a first robot control step of controlling the robot based on an operation command received from the operation interface; a first prediction simulation step of predicting interference between a model of the robot and a model of the surrounding environment in a simulation space based on status information received from a built-in sensor of the robot and an operation command received from the operation interface; a mode setting step of asking a user via the display device whether the predicted interference is an intended interference by the user, and if it is determined that the predicted interference is an intended interference, transitioning from the normal operation mode to the work operation mode, and if it is determined that the predicted interference is not an intended interference, continuing the normal operation mode; In the work operation mode, a manipulation amount adjusting step of generating an adjusted operation command from the operation command received from the operation interface; a second robot control step of controlling the robot based on the adjusted operation command; a second prediction simulation step of predicting interference between a model of the robot and a model of the surrounding environment in a simulation space based on status information received from a built-in sensor of the robot and the adjusted operation command.

Images