Robot control device, robot device, robot control system, and robot control method

Abstract

To provide a robot control device, a robot device, a robot control system, and a robot control method capable of appropriately operating the robot device in a changing environment.SOLUTION: A trajectory planning device 8 matches environment information acquired by an environment sensor 12 with an environment model which is a three-dimensional model of a surrounding environment of a robot device 1 to set a region from which the environment sensor has not been able to acquire environment information among regions adjacent to the environment model as a caution region, and plans an operation trajectory of a robot arm 10 on the basis of the target position and the caution region of the environment model and an end effector 11. A robot control device 7 operates the robot arm 10 on the basis of the operation trajectory.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a robot control system, a robot control device, a robot device, and a robot control method for controlling a robot that executes work.

Background Art

[0002] In order to use a robot in a changing environment, such as a mobile robot outdoors, the robot needs to move autonomously and perform work at the destination of movement, and it is necessary to cope with the deviation of its own position and the variation of the work object accompanying the movement. Therefore, there is known a robot that recognizes the surrounding environment of the robot using an environmental sensor such as a camera, and autonomously generates the operation of the robot based on the recognition result and performs the work. By estimating the position of the robot in the environment from the surrounding environment, estimating the position of the obstacle, avoiding the obstacle, and repeating the movement to the target position and performing the work, the robot device can perform a predetermined work even in an environment where the surroundings fluctuate or an environment where a previously unknown state exists.

[0003] As a prior art document regarding a control device for a robot that works autonomously, for example, there is Patent Document 1. Patent Document 1 discloses a control device including a detection unit that detects a change occurring in the surrounding environment of the working area of a robot arm, and changes the trajectory of the robot arm in the changed area. According to this control device, it is possible to perform work while avoiding contact between the robot arm and an obstacle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The control device of Patent Document 1 is configured to change the trajectory of the robotic arm in accordance with predetermined content according to the detection result of environmental changes. However, when the detection result of environmental changes is incomplete, there remains a possibility of contact with obstacles in the environment even if the trajectory of the robotic arm is changed, and there is a risk that the robotic arm may not be able to perform the work properly.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a robot control device, a robot device, a robot control system, and a robot control method capable of appropriately operating a robot device in a changing environment.

Means for Solving the Problems

[0007] In order to achieve the above object, the present invention provides a robot control device for controlling a robot device including a robotic arm having a rotating joint or a linearly moving joint and an end effector attached to the tip of the robotic arm, the robot control device setting a target position of the end effector, and planning an operation trajectory of the robotic arm based on an environmental model which is a three-dimensional model of the surrounding environment of the robot device and the target position, and including an environmental sensor for acquiring information on the surrounding environment as environmental information, the trajectory planning device matching the environmental information acquired by the environmental sensor with the environmental model, setting, as a caution area, an area in a region adjacent to the environmental model where the environmental sensor could not acquire environmental information, planning the operation trajectory based on the environmental model, the target position, and the caution area, and the robot control device operating the robotic arm based on the operation trajectory.

[0008] Further, the present invention provides a robot device including the robot control device, the robotic arm, and the end effector.

[0009] Further, the present invention provides a robot control system including the robot control device and the robot device.

[0010] Further, the present invention provides a robot control method for controlling a robot device including a robot arm having a rotating joint or a linearly moving joint and an end effector attached to the tip of the robot arm, the method comprising: a first step of acquiring environmental information around the robot device; a second step of matching the environmental information acquired in the first step with an environmental model which is a three-dimensional model of the environment around the robot device, and setting, as an attention area, an area in which the environmental information could not be acquired among areas adjacent to the environmental model; a third step of setting a target position of the end effector; a fourth step of planning an operation trajectory of the robot arm based on the environmental model, the target position, and the attention area; and a fifth step of operating the robot arm based on the operation trajectory.

Advantages of the Invention

[0011] According to the present invention, it is possible to appropriately operate a robot device in a changing environment.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Modes for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Examples

[0014] The first embodiment of the present invention will be described with reference to FIGS. 1 to 7.

[0015] Figure 1 is a schematic diagram showing the configuration of a robot control system according to a first embodiment of the present invention. The robot control system 100 includes a robot device 1, a robot control device 7, and a trajectory planning device 8. Note that the trajectory planning device 8 may be configured as a part of the robot control device 7 or as an independent device from the robot control device 7. Further, the robot control device 7 and the trajectory planning device 8 may be configured as a part of the robot device 1 or as an independent device from the robot device 1.

[0016] The robot device 1 has a mobile cart 9, a robot arm 10 attached to the mobile cart 9, and an end effector 11 attached to the tip of the robot arm 10. Each device of the robot device 1 is connected to the robot control device 7, receives a control command from the robot control device 7 (such as the motor current of the mobile cart 9, the motor current of the robot arm 10, etc.), operates, and transmits the state of the robot device 1 (such as the voltage of the angle sensor attached to the joint of the robot arm 10, etc.) to the robot control device 7.

[0017] The robot control device 7 is connected to the trajectory planning device 8 via a communication cable, converts the state of the robot device 1 obtained from the robot device 1 into digital data (such as the joint angle of the robot arm 10, the hand position of the end effector 11, etc.) and transmits it to the trajectory planning device 8, and is configured to calculate a control command for the robot device 1 based on the trajectory command output by the trajectory planning device 8 (such as the time-series data of the target joint angle of the robot arm 10, the time-series data of the target hand position of the end effector 11, etc.) and the state of the robot device 1 input from the robot device 1. Further, a camera 12 as an environmental sensor installed on the mobile cart 9 is connected to the trajectory planning device 8 and transmits an image of the surroundings of the robot device 1 captured by the camera to the trajectory planning device 8.

[0018] Figure 2 is a schematic diagram showing the hardware configuration for executing the software of the robot control system 100. Note that in Figure 2, the interface is described as "I / F".

[0019] The robot control device 7 is a computer in which a control device 71, a communication interface 72, a control interface 73, and a storage device 74 are electrically connected. The control device 71 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The storage device 74 is an auxiliary storage device such as a hard disk drive, and stores a control program 741 executed by the control device 71. The control interface 73 is an interface for connecting to the robot device 1 and transmitting and receiving control commands to the robot device 1 and data related to the state of the robot device 1, and is appropriately configured according to the devices constituting the robot device 1. The communication interface 72 is an interface for connecting to the trajectory planning device 8 and transmitting and receiving trajectory commands to the robot device 1 and data related to the state of the robot device 1. When the robot control device 7 is started by turning on the power or the like, it expands and executes the control program 741 stored in the storage device 74 in the control device 71. The control program 741 generates a control command to the robot device 1 based on the trajectory command of the trajectory planning device 8 input from the communication interface 72 and the state of the robot device 1 input from the control interface 73, outputs the control command from the control interface 73 to the robot device 1, and outputs the state of the robot device 1 input from the control interface 73 from the communication interface 72 to the trajectory planning device 8.

[0020] The trajectory planning device 8 is a computer to which a control device 81, a communication interface 82, an input device 83, and a storage device 84 are electrically connected. The control device 81 includes a CPU, a RAM, a ROM, etc., and is configured to execute information processing based on programs and various data. The communication interface 82 is an interface for connecting to the robot control device 7 and transmitting and receiving the trajectory command of the robot device 1 to the robot control device 7 and the state of the robot device 1. Further, the communication interface 82 communicates with the camera 12. The input device 83 is a device that captures input from a user such as a mouse or a keyboard, and controls the execution of the program of the trajectory planning device 8. The storage device 84 is an auxiliary storage device such as a hard disk drive, and stores an environment setting program 850, a trajectory planning program 860, environment data 841, work data 843, and trajectory setting data 844 that are executed by the control device 81. When the trajectory planning device 8 is started by turning on the power or the like, the environment setting program 850 and the trajectory planning program 860 stored in the storage device 84 are expanded to the control device 81. The environment setting program 850 sets the surrounding environment of the robot device 1 using the environment data 841 and the imaging data of the camera 12 input from the communication interface 82, and outputs it to the trajectory planning program 860. The trajectory planning program 860 generates a trajectory command for the robot device 1 based on the environment settings set by the environment setting program 850, the work data 843, and the trajectory setting data 844, and outputs the trajectory command from the communication interface 82 to the robot control device 7.

[0021] Note that the robot control device 7 and the trajectory planning device 8 may be configured with the same hardware, and the control program 741, the environment setting program 850, and the trajectory planning program 860 may be configured to be executed on the same hardware. The control interface 73 and the communication interface 72 may have the same configuration, and the robot control device 7 may be configured to control the robot device 1 via a network. Further, the connection between the robot control device 7 and the trajectory planning device 8 does not have to be directly connected by a communication cable, and may be configured to be indirectly connected via a network.

[0022] Next, a method in which the trajectory planning device 8 sets environmental information and the robot device 1 generates and outputs a trajectory command so as to execute a predetermined operation will be described with reference to FIGS. 3 to 7.

[0023] FIG. 3 is an external view showing an example of the working environment of the robot device 1. A working example of the robot device 1 will be described with reference to FIG. 3.

[0024] In the inspection and maintenance of facilities in a factory, a worker may go to the inspection point, perform visual inspection, and perform operations such as equipment operation and replacement as necessary. For example, as shown in FIG. 3, in order to inspect the piping facility 31, it may be necessary to access the inspection work positions 21 and 22. The piping facility 31 is suspended from the ceiling using a support 33, and different piping facilities 32 are fixed to the wall surface below the inspection work positions 21 and 22. In addition, equipment 34 used in the factory is installed on the wall surface.

[0025] In order for the robot device 1 to perform work appropriately in the environment as described above, it is necessary to move the end effector 11 of the robot device 1 to the inspection work positions 21 and 22 while avoiding the piping facilities 31 and 32. When the robot device 1 approaches the site shown in FIG. 3, it uses the camera 12 to recognize the surrounding environment, measures the position and posture of the robot device 1 on the surrounding environment, and uses environmental data 841 such as map data previously held to grasp the positions of obstacles on the surrounding environment, and plans a trajectory for accessing the inspection work positions 21 and 22 while avoiding contact. At this time, it is necessary to cope with the displacement of the position of the obstacle due to the error in position measurement and the undefined obstacle due to the lack of the held environmental data 841. For example, in an environment as shown in FIG. 3, the position of the piping facility 31 near the inspection work positions 21 and 22 may not be set accurately, or the position of the support 33 may not be recorded in the environmental data 841 and cannot be set. The robot device 1 needs to operate appropriately and perform the work even in such an environment.

[0026] FIG. 4 is a schematic diagram showing the software configuration of the environment setting program 850 and the trajectory planning program 860 executed by the trajectory planning device 8. FIG. 5 is a flowchart showing a method in which the trajectory planning device 8 generates a trajectory command for the robot device 1 and transmits it to the robot control device 7. FIG. 6 is a schematic diagram showing a method in which the environment setting program 850 sets the surrounding environment of the robot device 1. FIG. 7 is a schematic diagram showing a method in which the trajectory planning program 860 sets the operation area of the robot device 1 and plans a trajectory. FIG. 7(a) is a top view of the environment in FIG. 3, and FIG. 7(b) is a side view of the environment in FIG. 3. Using FIGS. 4 to 7, a method in which the trajectory planning device 8 generates a trajectory command for the robot device 1 will be described. Note that the environment setting program 850 and the trajectory planning program 860 are configured to include a space recognition unit 851, a target position setting unit 861, an operation area setting unit 862, and a trajectory planning unit 863 as software modules when deployed and executed in the control device 81.

[0027] The spatial recognition unit 851 acquires the image captured by the camera 12 in step S100, and then recognizes the surrounding environment of the robot device 1 using the map information and environmental construction parameters in the environmental data 841 in step S101, estimates the position of the robot device 1, and constructs a three-dimensional model of the surrounding environment. As shown in FIG. 6, positions where the luminance and color change rapidly in the image are recognized as feature points, and based on the change in the video due to the change in the distance and angle between the camera 12 and the object, the distance between the feature points and the camera 12 is calculated, and a three-dimensional point cloud is created using Visual SLAM (Simultaneous Localization and Mapping). Then, a three-dimensional environmental model is constructed by matching the created point cloud with the map information held in advance. At this time, the constructed environmental model is constructed by a recognition part that can be recognized by the camera 12 shown by the solid line in FIG. 7 and a non-recognition part that is not recognized but constructed from map information and the like shown by the broken line in FIG. 7. The spatial recognition unit 851 outputs environmental information including the environmental model constructed in this way to the trajectory planning program 860. Note that the method for estimating the position of the robot device 1 and constructing the three-dimensional model may be configured by a method based on different environmental sensors. For example, it is obvious that SLAM using LiDAR (Light Detection and Ranging) or SLAM using a depth camera may be used.

[0028] The target position setting unit 861 acquires work parameters that define work position information, work order, etc. from the work data 843 in step S102, and sets the target position to which the end effector 11 of the robot device 1 should move. Note that the method for setting the target position of the robot device 1 does not have to adopt a method of reading the work data stored in advance. For example, it is obvious that it may be configured to be given from an external control device via a network. Also, it may be configured to capture the image of the camera 12 and output the target position using a model learned to output the target position for the given image.

[0029] The operation area setting unit 862 takes in the environment information set by the environment setting program 850 in step S103, and classifies the surrounding environment of the robot device 1 into three types: an unrestricted area, a caution area, and a prohibited area as shown in FIG. 7. First, the operation area setting unit 862 divides the periphery of the robot device 1 into a grid pattern with a predetermined length as shown by the dotted line in FIG. 7. Subsequently, the area that overlaps with the environmental model set in the environmental information is set as the prohibited area. Subsequently, the area adjacent to the unrecognized part of the environmental model is set as the caution area. Also, assuming that there may be a support 33 above the piping facility 31, the entire upper part of the piping facility 31 is set as the caution area. Finally, the area that does not belong to the prohibited area and the caution area is set as the unrestricted area. Then, the classified areas are transmitted to the trajectory planning unit 863. Note that it is not necessary to adopt a method of classifying non - continuous areas divided into a grid pattern as shown in FIG. 7. It may be configured such that the constructed environmental model is set as the prohibited area, and a predetermined distance in the normal direction of the surface of the unrecognized part of the environmental model is continuously set as the caution area. Also, it is not necessary to limit the classification of areas to three types: an unrestricted area, a caution area, and a prohibited area. It is obvious that it may be configured to classify into two types: an area where operation is prohibited and an area without restrictions. Also, it may be configured to define different areas and classify them into four or more areas.

[0030] In step S104, the trajectory planning unit 863 plans the trajectory of the robot device 1 based on the target position set by the target position setting unit 102 in step S102 and the area classified by the operation area setting unit 862 in step S103. The trajectory planning unit 863 uses Transition-based RRT (Rapidly exploring random tree) to create trajectories for the mobile cart 9 and the robot arm 10. At this time, the cost is set to be higher in the attention area than in the unrestricted area. When moving the end effector 11 from the inspection work position 21 to the inspection work position 22 as shown by the dashed-dotted line in Fig. 7, when moving away from the wall surface from the inspection work position 21, move laterally to the side of the piping facility 31 while maintaining a certain distance from the piping facility 31, then pass under the piping facility 31 where the environmental model recognized by the camera 12 is clear, and finally move to the inspection work position 22 while maintaining a certain distance from the piping facility 31 on the side of the piping facility 31. The trajectory planning unit 863 uses the classified areas in this way to generate a trajectory such that the robot device 1 passes through the unrestricted area as much as possible. However, if it is impossible to reach the target position except by passing through the attention area, a trajectory passing through the attention area is generated. Note that the trajectory planning method is not necessarily limited to the method described above. For example, a potential function corresponding to the distance of the prohibited area or the attention area may be set, and the trajectory passing through the position with low potential may be calculated. It is obvious that different trajectory planning methods applying an optimization method for changing or restricting the operation of the robot arm 10 corresponding to the classified areas may also be used. Specific examples of methods for restricting the operation of the robot arm 10 in the attention area include prohibiting the entry of the robot arm 10 into the attention area and setting an upper limit on the translational speed or joint rotation speed of the robot arm 10 in the attention area, etc.

[0031] Finally, in step S105, the trajectory planning unit 863 outputs the generated trajectory of the robot device 1 to the robot control device 7 as a trajectory command and ends the process.

[0032] In this way, when the environmental information detected using an environmental sensor such as the camera 12 is incomplete, an area near where the acquired information of the environmental sensor is incomplete is classified, and based on the classified area, a trajectory of the robot device 1 is generated, thereby enabling an appropriate operation of the robot device 1 that does not come into contact with obstacles in the environment, and enabling the robot device 1 to perform an appropriate task even in a changing environment. Further, even when the environmental information becomes incomplete using a low-accuracy environmental sensor 12, an appropriate operation can be executed for the robot device 1, so that the cost reduction of the robot device 1 can be realized by using a low-cost environmental sensor 12.

[0033] (Summary) In the first embodiment, in a robot control device 7 that controls a robot device 1 including a robot arm 10 having a rotating joint or a translating joint and an end effector 11 attached to the tip of the robot arm 10, a target position of the end effector 11 is set, and based on an environmental model that is a three-dimensional model of the surrounding environment of the robot device 1 and the target position, a trajectory planning device 8 that plans an operation trajectory of the robot arm 10, and an environmental sensor 12 that acquires information on the surrounding environment as environmental information are provided. The trajectory planning device 8 matches the environmental information acquired by the environmental sensor 12 with the environmental model, and sets, as a caution area, an area in the area adjacent to the environmental model where the environmental sensor 12 could not acquire environmental information, and plans the operation trajectory based on the environmental model, the target position, and the caution area. The robot control device 7 operates the robot arm 10 based on the operation trajectory.

[0034] Further, the robot device 1 in the first embodiment includes a robot control device 7, a robot arm 10, and an end effector 11.

[0035] Further, the robot control system 100 in the first embodiment includes a robot control device 7 and a robot device 1.

[0036] Also, in the first embodiment, in a robot control method for controlling a robot device 1 including a robot arm 10 having a rotating joint or a translating joint and an end effector 11 attached to the tip of the robot arm 10, a first step of acquiring environmental information around the robot device 1, a second step of matching the environmental information acquired in the first step with an environmental model which is a three-dimensional model of the surrounding environment of the robot device 1, and setting, as a caution area, an area in which the environmental information could not be acquired among the areas adjacent to the environmental model, a third step of setting a target position of the end effector 11, a fourth step of planning an operation trajectory of the robot arm 10 based on the environmental model, the target position, and the caution area, and a fifth step of operating the robot arm 10 based on the operation trajectory are provided.

[0037] According to the first embodiment configured as described above, by planning the operation trajectory of the robot arm 10 in consideration of an area (caution area) in which the environmental sensor 12 could not acquire environmental information among the areas adjacent to the environmental model, the possibility that the robot device 1 contacts an obstacle can be reduced, so that the robot device 1 can be appropriately operated in a changing environment.

[0038] Also, the trajectory planning device 8 in the first embodiment plans the operation trajectory so that the operation of the robot arm 10 in the caution area is restricted. Thereby, the possibility that the robot device 1 contacts an obstacle can be further reduced.

Embodiment

[0039] As a second embodiment of the present invention, a method in which the trajectory planning device 8 sets an operation area based on the state of the acquired data of the environmental sensor 12 will be described with reference to FIG. 8.

[0040] FIG. 8 is a schematic diagram showing a method for the trajectory planning device 8 to classify the operation area of the robot device 1. FIG. 8(a) is a schematic diagram showing a classification method for an area with good detection accuracy, FIG. 8(b) is a schematic diagram showing a classification method for an area with poor detection accuracy, and FIG. 8(c) is a schematic diagram showing a classification method for an area in which an object not present in the environmental model is detected.

[0041] As described above, when Visual SLAM is applied to the acquired image of the camera 12, the acquired feature points can be output as a point cloud, and a high-precision point cloud can be acquired by using a depth image in addition. At this time, as shown in FIG. 8(a), the average of the lengths of the errors between the acquired point cloud and the matching result with the environmental model is calculated, and the area to which the attention area is applied is set to the range in which the length of the error is expanded by a predetermined amount from the environmental model. In the calculation of the error between the point cloud and the matching result of the environmental model, the density of the point cloud is calculated. As shown in the lower right of FIG. 8(a), when the density is equal to or less than a predetermined value, it is determined as an error value generated due to a malfunction during measurement and is not used in the matching with the environmental model or the calculation of the range of the attention area.

[0042] On the other hand, as shown in FIG. 8(b), the range to which the attention area is applied is expanded at a location where the error between the acquired point cloud and the matching result with the environmental model is large. As shown in FIG. 8(b), when the error is large, it is assumed that the position recognition is deviated or there is an error in the environmental map and the size of the environmental model is different. In that case, the attention area is enlarged so as to largely avoid such an area.

[0043] As shown in the lower right of FIG. 8(c), when there are point clouds that, although having a density equal to or higher than a predetermined value, cannot be matched with the environmental model, a region of a predetermined range including the corresponding point clouds is set as a caution area. As shown in FIG. 8(c), when there are point clouds in an area that does not exist in the environmental model, it is assumed that there are objects not defined in the environmental map, and in that case, the caution area is set so as to avoid such areas. Note that the setting of the caution area and the change of the range are not necessarily limited to the methods described above. For example, when a LiDAR is used as the environmental sensor, the reflection intensity of the laser is acquired, and the range of the caution area is set according to the reflection intensity, and the caution area may be configured to be changed according to the data acquisition status of the environmental sensor. Also, the method of setting the range of the area does not have to be a method of expanding a predetermined amount of the magnitude of the error. For example, it may be configured to change the range according to the attributes of the environmental model, and the set amount of the range may be variable.

[0044] By determining the classifiability of the area according to the state of the environmental sensor thus obtained, and changing the range of the area to be classified based on the degree of coincidence with the environmental model, it is possible to reduce the possibility of contact during the operation of the robot device 1, and the robot device 1 can perform an appropriate operation.

[0045] (Summary) In the trajectory planning device 8 in the second embodiment, the range of the caution area is changed according to the degree of coincidence between the environmental information acquired by the environmental sensor 12 and the environmental model.

[0046] According to the second embodiment configured as described above, since the caution area is set more appropriately, it is possible to further reduce the possibility that the robot device 1 contacts an obstacle.

[0047] Also, the trajectory planning device 8 in the second embodiment acquires the point cloud of the surrounding environment of the robot device 1 as the environmental information, calculates the average of the errors between the point cloud and the environmental model as the degree of coincidence, and sets a range obtained by expanding the average of the errors from the environmental model by a predetermined amount as the range of the caution area. Thereby, it becomes possible to set the caution area to a necessary and sufficient range.

Embodiment

[0048] As a third embodiment of the present invention, a method for planning a trajectory such that the trajectory planning device 8 changes the operation of the robot device 1 based on the classified area will be described with reference to FIG. 9.

[0049] FIG. 9 is a schematic diagram showing how the robot device 1 operates. The robot device 1 is provided with a hand camera 13 different from the camera 12, and is configured such that information in the vicinity of the end effector 11 can be acquired while the robot arm 10 is operating. The environmental model constructed in steps S100 to S101 shown in FIG. 5 cannot acquire the environmental data on the side of the piping facility 31 shown in FIG. 9, and an attention area is set on the side of the piping facility 31 as shown in FIG. 9 by the area setting executed in step S103 shown in FIG. 5. At this time, this attention area is an area where environmental data acquisition is incomplete, and it is desirable that environmental data of these areas can be acquired. The trajectory planning unit 863 plans a trajectory in step S104 shown in FIG. 5 so that the hand camera 13 faces this attention area. When the robot control device 7 executes the trajectory command planned in step S105, the hand camera 13 operates so as to face the attention area as shown in FIG. 9, and the hand camera 13 acquires information in the vicinity of the attention area. The acquired information may be configured to update the environmental data 841 stored in the storage device 84 of the trajectory planning device 8, or may be configured to be transmitted to the trajectory planning device 8 of a different robot device 1 via a network. Then, when the robot device 1 works again at the same work place, or when a different robot device 1 works at the same work place, a trajectory command is generated using the environmental data 841 updated by the acquired information.

[0050] In this way, by generating a trajectory command so as to acquire an area near where the information acquired by the environmental sensor 12 is incomplete in the measurement before generating the trajectory command, acquiring information during the operation of the robot device 1, and updating the environmental data, it is possible to reduce the possibility of contact during the operation in the subsequent work of the robot device 1 at the same work place, and the robot device 1 can perform appropriate work.

[0051] (Summary) The trajectory planning device 8 in the third embodiment plans the movement trajectory of the robot arm 10 by restricting the position or posture of the robot arm 10 so that the environmental sensor 13 can acquire environmental information within the attention area during the operation of the robot device 1.

[0052] According to the third embodiment configured as described above, since the environmental information of the attention area can be acquired during the operation of the robot device 1, it is possible to further reduce the possibility that the robot device 1 comes into contact with an obstacle.

Embodiment

[0053] As a fourth embodiment of the present invention, a method of determining an error in environmental information during the operation of the robot device 1 and adjusting a trajectory command at different work locations based on the magnitude of the error will be described with reference to FIG. 10.

[0054] FIG. 10 is a schematic diagram showing the software configuration of an environment setting program 850, a trajectory planning program 860, and an environment error determination program 870 executed by the trajectory planning device 8, as well as an environment state storage database 880. The robot device 1 is provided with a hand camera 13 which is an environment sensor different from the camera 12, and is configured to be able to acquire information in the vicinity of the end effector 11 while the robot arm 10 is operating. The environment error determination program constructs an environment model from the image acquired by the hand camera 13, compares the constructed environment model with the environment data 841, and estimates the magnitude of the error. Specifically, the 3D point cloud configured using the aforementioned Visual SLAM is compared with the map information in the environment data, and the average of the lengths of the displacements between the positions of the map data and the acquired 3D point cloud is calculated. The environment error determination program 870 transmits the magnitude of the error to the environment state storage database 880, and the environment state storage database 880 stores the magnitude of the error. When the magnitude of the error accumulated in the environment state storage database 880 is large, it can be determined that the accuracy of constructing the environment model by the camera 12 has decreased, or the accuracy of the environment data 841 has decreased, or there is a data deficiency. The trajectory planning program, when determining the region in step S103 shown in FIG. 5, reads the magnitude of the error stored in the environment state storage database 880, and adjusts the determination of the region to be classified according to the magnitude of the error. Specifically, in setting the attention region shown in FIG. 7, when the magnitude of the error stored in the environment state storage database 880 is equal to or greater than a predetermined value, the attention region is expanded not only to the region adjacent to the unrecognized part on the environment model but also to a range of two grids from the unrecognized part. Note that the error determination method may not be limited to the method described above. For example, it may be configured to count the amount of point clouds that cannot be matched with the environment model. Also, the data accumulated in the environment state storage database may not be limited to the magnitude of the error. The environment model and the magnitude of the error may be associated and accumulated, and the trajectory planning program 860 may be configured to adjust the determination of the region to be classified according to the current position.

[0055] In this way, by acquiring information on areas where the acquired information of the environmental sensor 12 is incomplete and accumulating the amount of error in the environmental data, it is possible to determine the accuracy of the environmental information at the workplace where the robot device 1 operates, and by adjusting the operation of the robot device 1 according to the accuracy of the environmental information, it is possible to reduce the possibility of contact during operation in the work, and the robot device 1 can perform appropriate work.

[0056] (Summary) The trajectory planning device 8 in the fourth embodiment stores the difference between the environmental information acquired by the environmental sensor 13 during the operation of the robot device 1 and the environmental model as an environmental error, and plans the operation trajectory of the robot arm 10 based on the environmental model, the target position of the end effector 11, the caution area, and the environmental error.

[0057] According to the fourth embodiment configured as described above, since the operation of the robot device 1 is adjusted according to the accuracy of the environmental information, it is possible to further reduce the possibility of the robot device 1 coming into contact with an obstacle.

Embodiment

[0058] As a fifth embodiment of the present invention, a method in which the area classified by the robot device 1 is transmitted to a remote operation device and the operator confirms and remotely operates the classified area will be described with reference to FIGS. 11 to 13.

[0059] FIG. 11 is a schematic diagram showing an example of a system configuration for executing the software of the robot control system 100, FIG. 12 is a schematic diagram showing the configuration of a remote operation device 15 for remotely operating the robot device 1, FIG. 13 is a schematic diagram showing the screen output of the remote operation device 15, FIG. 13(a) is a schematic diagram showing the screen output with the area classified in the image of the hand camera 13 superimposed, and FIG. 13(b) is a schematic diagram showing the screen output of an aerial view image three-dimensionally configured based on the environmental information configured by the trajectory planning device 8.

[0060] The robot device 1 is provided with a wireless communication device 14 for communicating with external devices. The trajectory planning device 8 transmits, via the communication interface 82, the images acquired by the camera 12 and the hand camera 13, and the information of the regions calculated and classified by the control device 81, receives operation commands from the outside, and is configured to output operation commands to the robot control device 7 based on the received information. The wireless communication device 14 is connected to a remote operation device 15 for remotely operating the robot device 1 shown in FIG. 12, transmits the information output by the robot device 1 to the remote operation device 15, and receives the operation commands output by the remote operation device 15.

[0061] The remote operation device 15 is a computer to which a control device 151, a communication interface 152, an input device 153, and a display device 154 are electrically connected. The control device 151 includes a CPU, a RAM, a ROM, etc., and is configured to execute information processing based on programs and various data. The communication interface 152 is an interface for connecting to the wireless communication device 14 and transmitting and receiving operation commands to / from the robot device 1 and the state of the robot device 1. The input device 153 is a device for taking in operation inputs from a user, such as a joystick and a keyboard shown in FIG. 12, and generates operation commands for the robot device 1. The display device 154 is a device for displaying the state of the robot device 1 to the user, such as a monitor shown in FIG. 12.

[0062] The robot device 1 autonomously performs operations based on the foregoing content. However, when it detects a state where the rotation angle of the motor of the robot arm 10 does not operate as commanded or the motor current becomes excessive, it determines that the operation cannot be performed and issues a request to the remote operation device 15 via the wireless communication device 14 to execute an operation. An operator (not shown) monitors the content of the display device 154 based on the request presented by the remote operation device 15 and operates the robot device 1 via the input device 153. At this time, as shown in FIG. 13(b), the display device 154 displays an overhead image configured three-dimensionally to present the overall state of the robot device 1 to the operator, and as shown in FIG. 13(a), it presents the captured image of the hand camera 13 to show the detailed state of the work site. The attention area may have an object that cannot be acquired by the environmental sensor. FIG. 13(a) shows that there are valves or damaged cables not included in the environmental map. The operator monitors the screen output of FIG. 13(a) and operates the robot device 1 to avoid objects that are not included in the environmental map within the attention area and cannot be acquired by the environmental sensor.

[0063] In this way, when the robot device 1 fails to appropriately perform an operation, when switching to manual operation by the operator, by presenting the classified areas, it is possible to draw the operator's attention during remote operation and enable the robot device to appropriately perform an operation during remote operation.

[0064] (Summary) The robot control system 100 in the fifth embodiment further includes a remote operation device 15 that remotely operates the robot device 1 and a wireless communication device 14 that communicates between the robot device 1 and the remote operation device 15. The wireless communication device 14 transmits the state of the robot device 1 and the environmental information that distinguishes the attention area from the area outside the attention area to the remote operation device 15. The remote operation device 15 displays the state of the robot device 1 and displays the environmental information by distinguishing the attention area from the area outside the attention area.

[0065] According to the fifth embodiment configured as described above, since the operator remotely operating the robot device 1 can grasp the state of the robot device 1 and the range of the caution area, it is possible to appropriately operate the robot device 1 by remote control.

[0066] Note that the present invention is not limited to the above-described embodiments and includes various modifications.

[0067] For example, although the robot device 1 used in the description of this embodiment is illustrated as being configured to move by the mobile cart 9 using wheels, it may be configured to move using a stage or move using legs. Also, although the robot arm 10 is illustrated as a vertically articulated robot, it may be an orthogonal coordinate type robot, a horizontally articulated robot, a parallel link robot, or the like.

[0068] The robot control device 7 is configured to operate in response to the trajectory command of the trajectory planning device 8, but it may be configured to receive different operation commands. For example, it may be configured to acquire the operation commands of the operator generated by lever operation or the like via a network and operate according to the commands, or it may be configured to be able to switch the commands to be acquired.

[0069] The environment setting program 850 reads the pre-stored environment data 841, but the environment data 841 may be configured to be acquired via a network. Also, although SLAM is used as the method for constructing the environment, NeRF (Neural Radiance Fields) using a model learned to construct a three-dimensional environment model from images may be used, and it is obvious that different methods for constructing an environment model using different environmental sensors may also be used.

Explanation of Reference Numerals

[0070] 1…Robot device, 7…Robot control device, 8…Trajectory planning device, 9…Mobile cart, 10…Robot arm, 11…End effector, 12…Camera (environmental sensor), 13…End effector camera (environmental sensor), 14…Wireless communication device, 15…Remote operation device, 21, 22…Inspection work positions, 33…Support, 34…Equipment, 71…Control device, 72…Communication interface, 73…Control interface, 74…Storage device, 81…Control device, 82…Communication interface, 83…Input device, 84…Storage device, 100…Robot control system, 102…Target position setting unit, 151…Control device, 152…Communication interface, 153…Input device, 154…Display device, 741…Control program, 841…Environmental data, 843…Work data, 844…Trajectory setting data, 850…Environmental setting program, 851…Space recognition unit, 860…Trajectory planning program, 861…Target position setting unit, 862…Operation area setting unit, 863…Trajectory planning unit, 870…Environmental error determination program, 880…Environmental state storage database, S100~S105…Steps.

Claims

1. In a robot control device for controlling a robot device including a robot arm having a rotating joint or a linearly moving joint and an end effector attached to the tip of the robot arm, an orbit planning device that sets a target position of the end effector and plans an operation orbit of the robot arm based on an environment model that is a three-dimensional model of the surrounding environment of the robot device and the target position; an environment sensor that acquires information on the surrounding environment as environment information; wherein the orbit planning device matches the environment information acquired by the environment sensor with the environment model, and sets, as a caution area, an area in the area adjacent to the environment model where the environment sensor could not acquire environment information; plans the operation orbit based on the environment model, the target position, and the caution area; the robot control device operates the robot arm based on the operation orbit A robot control device characterized by the above.

2. In the robot control device according to claim 1, the orbit planning device plans the operation orbit so that the operation of the robot arm in the caution area is restricted A robot control device characterized by the above.

3. In the robot control device according to claim 1, the orbit planning device changes the range of the caution area according to the degree of coincidence between the environment information acquired by the environment sensor and the environment model A robot control device characterized by the above.

4. In the robot control device according to claim 3, the orbit planning device acquires a point cloud of the surrounding environment as the environment information, calculates an average of errors between the point cloud and the environment model as the degree of coincidence, sets, as the range of the caution area, a range obtained by expanding the average of the errors from the environment model by a predetermined amount A robot control device characterized by the above.

5. In the robot control device according to claim 1, the orbit planning device plans the operation orbit by restricting the position or posture of the robot arm so that the environment sensor acquires environment information within the caution area during operation of the robot device A robot control device characterized by the above.

6. In the robot control device according to claim 1, the orbit planning device stores, as an environment error, a difference between the environment information acquired by the environment sensor during operation of the robot device and the environment model planning the motion trajectory based on the environmental model, the target position, the attention area, and the environmental error A robot control device characterized by the above.

7. The robot control device according to claim 1, the robot arm, and an end effector. A robot device characterized by the above.

8. The robot control device according to claim 1, and the robot device. A robot control system characterized by the above.

9. In the robot control system according to claim 8, a remote operation device for remotely operating the robot device, and a wireless communication device for communicating between the robot device and the remote operation device, wherein the wireless communication device transmits the state of the robot device and the environmental information that distinguishes the attention area from areas other than the attention area to the remote operation device, and the remote operation device displays the state of the robot device and displays the environmental information while distinguishing the attention area from areas other than the attention area. A robot control system characterized by the above.

10. In a robot control method for controlling a robot device including a robot arm having a rotating joint or a linearly moving joint and an end effector attached to the tip of the robot arm, a first step of acquiring environmental information around the robot device; a second step of matching the environmental information acquired in the first step with an environmental model that is a three-dimensional model of the environment around the robot device, and setting, as an attention area, an area in which the environmental information could not be acquired among areas adjacent to the environmental model; a third step of setting a target position of the end effector; a fourth step of planning a motion trajectory of the robot arm based on the environmental model, the target position, and the attention area; and a fifth step of operating the robot arm based on the motion trajectory. A robot control method characterized by the above.

Images