Robot control device, robot device, robot control system, and robot control method

Abstract

The present invention provides: a robot control device with which a robot device can be appropriately operated in a changing environment; a robot device; a robot control system; and a robot control method. For this purpose, a trajectory planning device (8) matches environment information acquired by an environment sensor (12) with an environment model, which is a three-dimensional model of the surrounding environment of a robot device (1), sets, as a caution region from among regions adjacent to the environment model, a region in which the environment sensor was unable to acquire the environment information, and plans an operation trajectory of a robot arm (10) on the basis of the environment model, the target position of an end effector (11), and the caution region. A robot control device (7) operates the robot arm (10) on the basis of the operation trajectory.

Description

Robot control device, robot device, robot control system, and robot control method

[0001] The present invention relates to a robot control system, a robot control device, a robot device, and a robot control method for controlling a robot that performs a task.

[0002] To use a robot in a changing environment, such as an outdoor mobile robot, the robot must be able to move autonomously and perform tasks at the new location, and must be able to deal with shifts in its own position and changes in the target object that accompany the movement. Therefore, robots that recognize the environment around the robot using environmental sensors such as cameras and autonomously generate robot actions based on the recognition results to perform tasks are known. By repeatedly estimating the robot's position in the environment from the surrounding environment, estimating the positions of obstacles, avoiding the obstacles, and moving to the target location to perform the task, the robot can perform a predetermined task even in a changing environment or an environment where the surroundings are unknown.

[0003] Prior art documents relating to control devices for autonomously working robots include, for example, Patent Document 1. Patent Document 1 discloses a control device that includes a detection unit that detects changes in the surrounding environment of the working area of ​​the robot arm and changes the trajectory of the robot arm in the changed area. This control device enables the robot arm to perform work while avoiding contact with obstacles.

[0004] Patent No. 6508691

[0005] The control device in Patent Document 1 is configured to change the trajectory of the robot arm in accordance with predetermined content depending on the results of detecting environmental changes, but if the results of detecting environmental changes are incomplete, even if the trajectory of the robot arm is changed, there is still a possibility that it will come into contact with an obstacle in the environment, and the robot arm may not be able to perform its work properly.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its object is to provide a robot control device, a robot device, a robot control system, and a robot control method that are capable of operating a robot device appropriately in a changing environment.

[0007] In order to achieve the above object, the present invention provides a robot control device for controlling a robot device equipped with a robot arm having a rotating joint or a joint that moves in parallel with the robot arm and an end effector attached to the tip of the robot arm, the device comprising: a trajectory planning device that sets a target position for the end effector and plans a motion trajectory of the robot arm based on the target position and an environmental model that is a three-dimensional model of the environment surrounding the robot device; and an environmental sensor that acquires information about the surrounding environment as environmental information, the trajectory planning device matches the environmental information acquired by the environmental sensor with the environmental model, and sets, as an attention area, an area adjacent to the environmental model from which the environmental sensor was unable to acquire environmental information, and plans the motion trajectory based on the environmental model, the target position, and the attention area, and the robot control device operates the robot arm based on the motion trajectory.

[0008] Furthermore, the present invention provides a robot device comprising the robot control device, the robot arm, and the end effector.

[0009] Furthermore, the present invention provides a robot control system including the robot control device and the robot device.

[0010] The present invention also provides a robot control method for controlling a robot device including a robot arm having a rotating joint or a joint that moves in parallel with the robot arm, and an end effector attached to the tip of the robot arm, the method comprising: a first step of acquiring environmental information about the surroundings of the robot device; a second step of matching the environmental information acquired in the first step with an environmental model that is a three-dimensional model of the surrounding environment of the robot device, and setting, as an attention area, an area adjacent to the environmental model from which the environmental information could not be acquired; a third step of setting a target position for the end effector; a fourth step of planning a motion trajectory of the robot arm based on the environmental model, the target position, and the attention area; and a fifth step of operating the robot arm based on the motion trajectory.

[0011] According to the present invention, it is possible to make a robot device operate appropriately in a changing environment.

[0012] FIG. 1 is a schematic diagram showing the configuration of a robot control system in a first embodiment of the present invention. FIG. 1 is a schematic diagram showing the hardware configuration for executing software of the robot control system in the first embodiment of the present invention. FIG. 2 is an external view showing an example of a working environment of a robot device in the first embodiment of the present invention. FIG. 3 is a schematic diagram showing the software configuration of an environment setting program and a trajectory planning program executed by a trajectory planning device in the first embodiment of the present invention. FIG. 4 is a flowchart showing a method by which a trajectory planning device in the first embodiment of the present invention generates a trajectory command for a robot device and transmits it to a robot control device. FIG. 5 is a schematic diagram showing a method by which an environment setting program in the first embodiment of the present invention sets the surrounding environment of a robot device. FIG. 6 is a schematic diagram showing a method by which a trajectory planning program in the first embodiment of the present invention sets a motion area of ​​a robot device and plans a trajectory. FIG. 7 is a schematic diagram showing a method by which a trajectory planning program in the first embodiment of the present invention sets a motion area of ​​a robot device and plans a trajectory. FIG. 8 is a schematic diagram showing a method by which a trajectory planning device in the second embodiment of the present invention classifies motion areas of a robot device. FIG. 9 is a schematic diagram showing a method by which a trajectory planning device in the second embodiment of the present invention classifies motion areas of a robot device. FIG. 10 is a schematic diagram showing a manner in which a robot device in a third embodiment of the present invention operates. FIG. 10 is a schematic diagram showing the software configuration of an environment setting program, a trajectory planning program, and an environmental error determination program executed by a trajectory planning device according to a fourth embodiment of the present invention, as well as an environmental state storage database. FIG. 11 is a schematic diagram showing a hardware configuration for executing software for a robot control system according to a fifth embodiment of the present invention. FIG. 12 is a schematic diagram showing the configuration of a remote control device that remotely operates a robot device according to the fifth embodiment of the present invention. FIG. 13 is a schematic diagram showing screen output of a remote control device according to the fifth embodiment of the present invention.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] A first embodiment of the present invention will be described with reference to FIGS. 1 to 7B.

[0015] 1 is a schematic diagram showing the configuration of a robot control system according to a first embodiment of the present invention. The robot control system 100 includes a robot device 1, a robot control device 7, and a trajectory planning device 8. The trajectory planning device 8 may be configured as part of the robot control device 7, or may be configured as a device independent of the robot control device 7. The robot control device 7 and the trajectory planning device 8 may be configured as part of the robot device 1, or may be configured as a device independent of the robot device 1.

[0016] The robot device 1 has a mobile carriage 9, a robot arm 10 attached to the mobile carriage 9, and an end effector 11 attached to the tip of the robot arm 10. Each device of the robot device 1 is connected to a robot control device 7, and operates in response to control commands from the robot control device 7 (motor current of the mobile carriage 9, motor current of the robot arm 10, etc.), and also transmits the state of the robot device 1 (voltage of an angle sensor attached to a joint of the robot arm 10, etc.) to the robot control device 7.

[0017] The robot control device 7 is connected to the trajectory planning device 8 via a communication cable, and is configured to convert the state of the robot device 1 obtained from the robot device 1 into digital data (joint angles of the robot arm 10, hand position of the end effector 11, etc.) and transmit it to the trajectory planning device 8, and to calculate control commands for the robot device 1 based on the trajectory commands output by the trajectory planning device 8 (time series data of the target joint angles of the robot arm 10, time series data of the target hand position of the end effector 11, etc.) and the state of the robot device 1 input from the robot device 1. In addition, a camera 12 as an environmental sensor installed on the mobile carriage 9 is connected to the trajectory planning device 8, and transmits captured images of the surroundings of the robot device 1 to the trajectory planning device 8.

[0018] Fig. 2 is a schematic diagram showing the hardware configuration that executes the software of the robot control system 100. Note that in Fig. 2, the interface is written as "I / F".

[0019] The robot control device 7 is a computer electrically connected to a control device 71, a communication interface 72, a control interface 73, and a storage device 74. The control device 71 includes a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), etc., and is configured to execute information processing based on programs and various data. The storage device 74 is an auxiliary storage device such as a hard disk drive, and stores a control program 741 executed by the control device 71. The control interface 73 is an interface connected to the robot device 1 and used to send and receive control commands to the robot device 1 and data related to the status of the robot device 1, and is configured appropriately depending on the components constituting the robot device 1. The communication interface 72 is an interface connected to the trajectory planning device 8 and used to send and receive trajectory commands to the robot device 1 and data related to the status of the robot device 1. When the robot control device 7 is started, for example, by powering on, the control program 741 stored in the storage device 74 is loaded into the control device 71 and executed. The control program 741 generates a control command for the robot device 1 based on the trajectory command of the trajectory planning device 8 input from the communication interface 72 and the state of the robot device 1 input from the control interface 73, outputs the control command from the control interface 73 to the robot device 1, and outputs the state of the robot device 1 input from the control interface 73 to the trajectory planning device 8 from the communication interface 72.

[0020] The trajectory planning device 8 is a computer electrically connected to a control device 81, a communication interface 82, an input device 83, and a storage device 84. The control device 81 includes a CPU, RAM, ROM, etc., and is configured to execute information processing based on programs and various data. The communication interface 82 is connected to the robot control device 7 and is an interface for transmitting and receiving trajectory commands for the robot device 1 and the status of the robot device 1 to the robot control device 7. The communication interface 82 also communicates with the camera 12. The input device 83 is a device for receiving input from a user, such as a mouse or keyboard, and controls the execution of the program of the trajectory planning device 8. The storage device 84 is an auxiliary storage device such as a hard disk drive, and stores an environment setting program 850, a trajectory planning program 860, environment data 841, work data 843, and trajectory setting data 844 executed by the control device 81. When the trajectory planning device 8 is started by turning on the power, etc., it loads the environment setting program 850 and the trajectory planning program 860 stored in the storage device 84 into the control device 81. The environment setting program 850 sets the surrounding environment of the robot apparatus 1 using the environment data 841 and the image data of the camera 12 input from the communication interface 82, and outputs the set environment to the trajectory planning program 860. The trajectory planning program 860 generates a trajectory command for the robot apparatus 1 based on the environment settings set by the environment setting program 850, the work data 843, and the trajectory setting data 844, and outputs the trajectory command to the robot control device 7 from the communication interface 82.

[0021] The robot control device 7 and the trajectory planning device 8 may be configured with the same hardware, and the control program 741, the environment setting program 850, and the trajectory planning program 860 may be configured to be executed on the same hardware. The control interface 73 and the communication interface 72 may have the same configuration, and the robot control device 7 may be configured to control the robot device 1 via a network. Furthermore, the robot control device 7 and the trajectory planning device 8 do not have to be directly connected by a communication cable, and may be indirectly connected via a network.

[0022] Next, a method in which the trajectory planning device 8 sets environmental information and generates and outputs a trajectory command so that the robot device 1 executes a predetermined task will be described with reference to FIGS. 3 to 7B.

[0023] 3 is an external view showing an example of a working environment of the robot device 1. An example of a work performed by the robot device 1 will be described with reference to FIG.

[0024] During the inspection and maintenance of equipment in a factory, workers may visit inspection locations to perform visual inspections and, if necessary, operate or replace equipment. For example, as shown in Fig. 3, inspection work positions 21 and 22 may be accessed to inspect piping equipment 31. The piping equipment 31 is suspended from the ceiling using supports 33, and different piping equipment 32 is fixed to the wall below the inspection work positions 21 and 22. In addition, equipment 34 used in the factory is installed on the wall.

[0025] In order for the robot device 1 to perform its work appropriately in the above-described environment, it is necessary to move the end effector 11 of the robot device 1 to the inspection work positions 21 and 22 while avoiding the piping equipment 31 and 32. When the robot device 1 approaches the work site shown in FIG. 3 , it uses the camera 12 to recognize the surrounding environment, measures the position and orientation of the robot device 1 in the surrounding environment, and uses pre-stored environmental data 841, such as map data, to identify the locations of obstacles in the surrounding environment and plan a trajectory to access the inspection work positions 21 and 22 while avoiding contact. During this process, it is necessary to deal with position shifts of obstacles due to position measurement errors and undefined obstacles due to incompleteness of the stored environmental data 841. For example, in the environment shown in FIG. 3 , it may be impossible to accurately set the position of the piping equipment 31 near the inspection work positions 21 and 22, or the position of the support 33 may not be recorded in the environmental data 841 and therefore cannot be set. Therefore, the robot device 1 must be able to operate appropriately and perform its work even in such an environment.

[0026] FIG. 4 is a schematic diagram showing the software configuration of the environment setting program 850 and the trajectory planning program 860 executed by the trajectory planning device 8. FIG. 5 is a flowchart showing a method by which the trajectory planning device 8 generates a trajectory command for the robot device 1 and transmits it to the robot control device 7. FIG. 6 is a schematic diagram showing a method by which the environment setting program 850 sets the surrounding environment of the robot device 1. FIGS. 7A and 7B are schematic diagrams showing a method by which the trajectory planning program 860 sets the operating area of ​​the robot device 1 and plans a trajectory. FIG. 7A is a top view of the environment of FIG. 3, and FIG. 7B is a side view of the environment of FIG. 3. The method by which the trajectory planning device 8 generates a trajectory command for the robot device 1 will be described using FIGS. 4 to 7B. The environment setting program 850 and the trajectory planning program 860 are configured to include, as software modules, a space recognition unit 851, a target position setting unit 861, an operating area setting unit 862, and a trajectory planning unit 863 when deployed and executed in the control device 81.

[0027] In step S100, the spatial recognition unit 851 acquires an image captured by the camera 12. Then, in step S101, the spatial recognition unit 851 recognizes the environment surrounding the robot device 1 using the map information and environment construction parameters in the environment data 841, estimates the position of the robot device 1, and constructs a three-dimensional model of the environment. As shown in FIG. 6 , locations in the image where brightness or color changes suddenly are recognized as feature points. A three-dimensional point cloud is created using Visual SLAM (Simultaneous Localization and Mapping), which calculates the distance between the feature points and the camera 12 based on changes in the image caused by changes in the distance and angle between the camera 12 and an object. The created point cloud is then matched with pre-stored map information to construct a three-dimensional environmental model. The constructed environmental model is constructed from a recognition section, shown by the solid lines in FIGS. 7A and 7B , that has been recognized by the camera 12, and a non-recognition section, shown by the dashed lines in FIGS. 7A and 7B , that has not been recognized but has been constructed based on map information or the like. The spatial recognition unit 851 outputs environmental information, including the constructed environmental model, to the trajectory planning program 860. It is clear that the method for estimating the position of the robot device 1 and constructing the 3D model may be based on a different environmental sensor, such as SLAM using LiDAR (Light Detection and Ranging) or SLAM using a depth camera.

[0028] In step S102, the target position setting unit 861 acquires task parameters that define task position information, task sequence, etc. from the task data 843, and sets a target position to which the end effector 11 of the robot device 1 should move. It is clear that the method for setting the target position of the robot device 1 does not have to be a method of reading pre-stored task data, and the target position may be provided from an external control device via a network, for example. Furthermore, the target position may be output by capturing an image from the camera 12 using a model trained to output a target position for a given image.

[0029] The operation area setting unit 862 imports the environmental information set by the environment setting program 850 in step S103 and classifies the environment surrounding the robot apparatus 1 into three types based on the environmental information: unlimited area, caution area, and prohibited area, as shown in Figures 7A and 7B. The operation area setting unit 862 first divides the periphery of the robot apparatus 1 into a grid of a predetermined length, as shown by the dotted lines in Figures 7A and 7B. Next, the operation area setting unit 862 sets areas that overlap with the environmental model set in the environmental information as prohibited areas. Next, the operation area setting unit 862 sets areas adjacent to non-recognized parts of the environmental model as caution areas. Furthermore, since there is a possibility that supports 33 may be present above the piping equipment 31, the operation area setting unit 862 sets the entire area above the piping equipment 31 as a caution area. Finally, the operation area setting unit 862 sets areas that do not belong to the prohibited area or caution area as unlimited area. The operation area setting unit 862 then transmits the classified areas to the trajectory planning unit 863. 7A and 7B, it is not necessary to adopt the method of classifying discontinuous regions divided into a grid, but rather it is possible to set the constructed environmental model as a prohibited region and set a predetermined distance in the normal direction of the surface of the non-recognition part of the environmental model as a continuous attention region. Furthermore, it is clear that the classification of regions does not have to be limited to three: unrestricted region, attention region, and prohibited region, but can be classified into two: a region where movement is prohibited and a region without restrictions, or different regions can be defined and classified into four or more regions.

[0030] In step S104, the trajectory planning unit 863 plans a trajectory for the robot device 1 based on the target position set by the target position setting unit 861 in step S102 and the area classified by the operating area setting unit 862 in step S103. The trajectory planning unit 863 uses a transition-based RRT (rapidly exploring random tree) to create trajectories for the mobile carriage 9 and the robot arm 10. At this time, the cost is set to be higher in the attention area compared to the unlimited area. When the end effector 11 moves from the inspection work position 21 to the inspection work position 22 as shown by the dashed dotted lines in FIGS. 7A and 7B , when moving away from the wall surface from the inspection work position 21, it moves beside the piping equipment 31 while maintaining a certain distance from the piping equipment 31, then passes under the piping equipment 31, which can be recognized by the camera 12 and has a clear environmental model, and finally moves beside the piping equipment 31 to the inspection work position 22 while maintaining a certain distance from the piping equipment 31. The trajectory planning unit 863 uses the classified regions to generate a trajectory for the robot device 1 so that it passes through the unlimited region as much as possible, but if the target position can only be reached by passing through the attention region, it generates a trajectory that passes through the attention region. Note that the trajectory planning method is not limited to the above-described method. For example, a potential function may be set according to the distance of the prohibited region or the attention region, and a trajectory that passes through a position with low potential may be calculated. It is clear that a different trajectory planning method that applies an optimization method that changes or restricts the movement of the robot arm 10 in accordance with the classified region may also be used. Specific examples of methods for restricting the movement of the robot arm 10 in the attention region include prohibiting the robot arm 10 from entering the attention region, or setting upper limits on the translation speed or joint rotation speed of the robot arm 10 in the attention region.

[0031] Finally, in step S105, the trajectory planning unit 863 outputs the generated trajectory of the robot device 1 to the robot control device 7 as a trajectory command, and the process ends.

[0032] In this way, when the environmental information detected using an environmental sensor such as the camera 12 is incomplete, the nearby area where the information acquired by the environmental sensor is incomplete is classified, and a trajectory of the robot device 1 is generated based on the classified area, thereby generating an appropriate movement of the robot device 1 that does not come into contact with obstacles in the environment, and allowing the robot device 1 to perform an appropriate task even in a changing environment. Furthermore, because appropriate movement can be performed on the robot device 1 even when the environmental information is incomplete using a low-precision environmental sensor 12, the cost of the robot device 1 can be reduced by using a low-cost environmental sensor 12.

[0033] (Summary) In the first embodiment, a robot control device 7 controls a robot device 1 including a robot arm 10 having a rotating joint or a joint that moves in parallel with the robot arm 10, and an end effector 11 attached to the tip of the robot arm 10. The robot control device 7 includes a trajectory planning device 8 that sets a target position for the end effector 11 and plans a motion trajectory of the robot arm 10 based on an environmental model that is a three-dimensional model of the environment surrounding the robot device 1 and the target position, and an environmental sensor 12 that acquires information about the environment as environmental information. The trajectory planning device 8 matches the environmental information acquired by the environmental sensor 12 with the environmental model, and sets, as an attention area, an area adjacent to the environmental model from which the environmental sensor 12 was unable to acquire environmental information. The trajectory planning device 8 plans the motion trajectory based on the environmental model, the target position, and the attention area, and the robot control device 7 operates the robot arm 10 based on the motion trajectory.

[0034] The robot device 1 in the first embodiment also includes a robot control device 7 , a robot arm 10 , and an end effector 11 .

[0035] The robot control system 100 in the first embodiment also includes a robot control device 7 and a robot device 1 .

[0036] Moreover, in the first embodiment, a robot control method for controlling a robot device 1 including a robot arm 10 having a joint that rotates or a joint that moves in parallel and an end effector 11 attached to the tip of the robot arm 10 includes the following steps: a first step of acquiring environmental information about the surroundings of the robot device 1; a second step of matching the environmental information acquired in the first step with an environmental model that is a three-dimensional model of the surrounding environment of the robot device 1, and setting, as an attention area, an area adjacent to the environmental model for which the environmental information could not be acquired; a third step of setting a target position for the end effector 11; a fourth step of planning a motion trajectory for the robot arm 10 based on the environmental model, the target position, and the attention area; and a fifth step of operating the robot arm 10 based on the motion trajectory.

[0037] According to the first embodiment configured as described above, by planning the movement trajectory of the robot arm 10 taking into consideration the area (attention area) adjacent to the environmental model where the environmental sensor 12 was unable to acquire environmental information, the possibility of the robot device 1 coming into contact with an obstacle can be reduced, and the robot device 1 can be made to operate appropriately in a changing environment.

[0038] Furthermore, the trajectory planning device 8 in the first embodiment plans the movement trajectory so that the movement of the robot arm 10 in the attention area is restricted, thereby further reducing the possibility that the robot device 1 will come into contact with an obstacle.

[0039] As a second embodiment of the present invention, a method in which the trajectory planning device 8 sets an operating region based on the state of data acquired by the environmental sensor 12 will be described with reference to FIGS. 8A to 8C.

[0040] 8A to 8C are schematic diagrams showing how the trajectory planning device 8 classifies the operating areas of the robot device 1, where FIG. 8A is a schematic diagram showing a method for classifying areas with good detection accuracy, FIG. 8B is a schematic diagram showing a method for classifying areas with poor detection accuracy, and FIG. 8C is a schematic diagram showing a method for classifying areas when an object not in the environmental model is detected.

[0041] As described above, applying Visual SLAM to the image acquired by the camera 12 makes it possible to output the acquired feature points as a point cloud, and by using a depth image, it is possible to acquire a high-resolution point cloud. In this case, as shown in FIG. 8A, the average length of the error between the acquired point cloud and the matching result of the environmental model is calculated, and the area to which the attention area is applied is set as a range obtained by expanding the error length by a predetermined amount from the environmental model. Note that, when calculating the error between the point cloud and the environmental model, the density of the point cloud is calculated. If the density is below a predetermined value, as shown in the lower right of FIG. 8A, it is determined to be an error value caused by a measurement malfunction and is not used in matching with the environmental model or calculating the range of the attention area.

[0042] On the other hand, in areas where there is a large error in the matching results between the acquired point cloud and the environment model, the area to which the attention region is applied is expanded as shown in Fig. 8B. When the error is large as shown in Fig. 8B, it is assumed that the position recognition is off or there is an error in the environment map, causing the size of the environment model to differ. In such cases, the attention region is enlarged to avoid such areas.

[0043] As shown in the lower right of FIG. 8C , if there is a point cloud whose density is equal to or greater than a predetermined value but cannot be matched with the environmental model, a predetermined range of area including the relevant point cloud is set as an attention area. As shown in FIG. 8C , if there is a point cloud in an area that does not exist in the environmental model, it is assumed that there is an object not defined in the environmental map. In such a case, an attention area is set to avoid such an area. Note that the setting and change of the range of the attention area are not limited to the above-described method. For example, if a LiDAR is used as the environmental sensor, the attention area may be changed depending on the data acquisition status of the environmental sensor, such as by acquiring the laser reflection intensity and setting the range of the attention area according to the reflection intensity. Furthermore, the method for setting the range of the area does not have to be a method of expanding the magnitude of the error by a predetermined amount. For example, the range may be changed depending on the attributes of the environmental model, or the range setting amount may be variable.

[0044] By determining whether or not an area can be classified based on the state of the environmental sensor acquired in this way, and by changing the range of the area to be classified based on the degree of agreement with the environmental model, the possibility of contact during operation of the robot device 1 can be reduced, allowing the robot device 1 to perform appropriate tasks.

[0045] (Summary) The trajectory planning device 8 in the second embodiment changes the range of the attention area depending on the degree of agreement between the environmental information acquired by the environmental sensor 12 and the environmental model.

[0046] According to the second embodiment configured as described above, the attention area is set more appropriately, so that it is possible to further reduce the possibility that the robot apparatus 1 will come into contact with an obstacle.

[0047] Furthermore, the trajectory planning device 8 in the second embodiment acquires a point cloud of the surrounding environment of the robot device 1 as the environmental information, calculates the average error between the point cloud and the environmental model as the degree of match, and sets the range of the attention region as a range obtained by expanding the average error from the environmental model by a predetermined amount, thereby making it possible to set the attention region to a necessary and sufficient range.

[0048] As a third embodiment of the present invention, a method for planning a trajectory by the trajectory planning device 8 so as to change the operation of the robot device 1 based on the classified region will be described with reference to FIG.

[0049] FIG. 9 is a schematic diagram showing the operation of the robot device 1. The robot device 1 includes a hand camera 13, separate from the camera 12, and is configured to acquire information about the vicinity of the end effector 11 while the robot arm 10 is operating. The environmental model constructed in steps S100 and S101 of FIG. 5 does not acquire environmental data about the area to the side of the piping equipment 31 shown in FIG. 9. Therefore, an attention area is set to the side of the piping equipment 31 as shown in FIG. 9 by the area setting executed in step S103 of FIG. 5. At this time, this attention area is an area where environmental data acquisition is incomplete, and it is desirable to acquire environmental data about this area. In step S104 of FIG. 5, the trajectory planning unit 863 plans a trajectory so that the hand camera 13 faces the attention area. Then, when the robot control device 7 executes the trajectory command planned in step S105, the hand camera 13 operates to face the attention area as shown in FIG. 9, and acquires information about the area to the side. The acquired information may be configured to update the environmental data 841 stored in the storage device 84 of the trajectory planning device 8, or may be configured to be transmitted via a network to the trajectory planning device 8 of a different robot device 1. Then, when the robot device 1 works again in the same work location, or when a different robot device 1 works in the same work location, a trajectory command is generated using the environmental data 841 updated with the acquired information.

[0050] In this way, by generating a trajectory command to acquire nearby areas where the information acquired by the environmental sensor 12 was incomplete in the measurement before the trajectory command was generated, acquiring the information while the robot device 1 is operating, and updating the environmental data, it is possible to reduce the possibility of contact during operation when the robot device 1 subsequently works at the same work location, and to allow the robot device 1 to perform appropriate work.

[0051] (Summary) The trajectory planning device 8 in the third embodiment plans the motion trajectory of the robot arm 10 by restricting the position or posture of the robot arm 10 so that the environmental sensor 13 acquires environmental information within the attention area while the robot device 1 is operating.

[0052] According to the third embodiment configured as described above, environmental information of the attention area can be acquired while the robot apparatus 1 is operating, which further reduces the possibility of the robot apparatus 1 coming into contact with an obstacle.

[0053] As a fourth embodiment of the present invention, a method for determining an error in environmental information while the robot apparatus 1 is operating and adjusting a trajectory command for a different work location based on the magnitude of the error will be described with reference to FIG.

[0054] FIG. 10 is a schematic diagram showing the software configuration of the environment setting program 850, trajectory planning program 860, environmental error determination program 870, and environmental state storage database 880 executed by the trajectory planning device 8. The robot device 1 is equipped with a hand-held camera 13, an environmental sensor different from the camera 12, which is configured to acquire information about the vicinity of the end effector 11 while the robot arm 10 is operating. The environmental error determination program constructs an environmental model from images acquired by the hand-held camera 13 and compares the constructed environmental model with environmental data 841 to estimate the magnitude of the error. Specifically, it compares a 3D point cloud constructed using the aforementioned Visual SLAM with the map information in the environmental data and calculates the average length of the deviation between the position in the map data and the acquired 3D point cloud. The environmental error determination program 870 transmits the magnitude of the error to the environmental state storage database 880, which stores the magnitude of the error. If the magnitude of the error stored in the environmental state storage database 880 is large, it can be determined that the accuracy of the construction of the environmental model by the camera 12 has decreased, the accuracy of the environmental data 841 has decreased, or there is a data deficiency. When performing area determination in step S103 shown in FIG. 5 , the trajectory planning program reads the magnitude of the error stored in the environmental state storage database 880 and adjusts the determination of the area to be classified based on the magnitude of the error. Specifically, in the setting of the attention area shown in FIGS. 7A and 7B , if the magnitude of the error stored in the environmental state storage database 880 is equal to or greater than a predetermined value, the attention area is expanded not only to the area adjacent to the unrecognized portion of the environmental model, but also to a range of two grids from the unrecognized portion. Note that the error determination method is not limited to the above-described method. For example, it may be configured to count the number of points that do not match the environmental model. Furthermore, the data stored in the environmental state storage database need not be limited to the magnitude of the error. The data may be stored in association with the environmental model and the magnitude of the error, and the trajectory planning program 860 adjusts the determination of the area to be classified based on the current position.

[0055] In this way, by acquiring information on areas where the information acquired by the environmental sensor 12 is incomplete and accumulating the amount of error in the environmental data, it is possible to determine the accuracy of the environmental information in the work area where the robot device 1 is operating, and by adjusting the operation of the robot device 1 according to the accuracy of the environmental information, it is possible to reduce the possibility of contact during operation, and to allow the robot device 1 to perform appropriate work.

[0056] (Summary) The trajectory planning device 8 in the fourth embodiment stores the difference between the environmental information acquired by the environmental sensor 13 during operation of the robot device 1 and the environmental model as an environmental error, and plans the operating trajectory of the robot arm 10 based on the environmental model, the target position of the end effector 11, the attention area, and the environmental error.

[0057] According to the fourth embodiment configured as described above, the operation of the robot apparatus 1 is adjusted depending on the accuracy of the environmental information, which further reduces the possibility of the robot apparatus 1 coming into contact with an obstacle.

[0058] As a fifth embodiment of the present invention, a method in which the robot device 1 transmits the classified areas to the remote control device, and the operator checks the classified areas and remotely controls the robot device 1 will be described with reference to FIGS. 11 to 13B.

[0059] FIG. 11 is a schematic diagram showing an example of a system configuration for executing software of the robot control system 100, FIG. 12 is a schematic diagram showing the configuration of a remote control device 15 that remotely controls the robot device 1, FIGS. 13A and 13B are schematic diagrams showing the screen output of the remote control device 15, FIG. 13A is a schematic diagram showing the screen output in which classified areas are superimposed on the image of the hand camera 13, and FIG. 13B is a schematic diagram showing the screen output of an overhead image constructed three-dimensionally based on environmental information constructed by the trajectory planning device 8.

[0060] The robot apparatus 1 is equipped with a wireless communication device 14 for communicating with external devices, and the trajectory planning device 8 is configured to transmit images acquired by the camera 12 and the hand camera 13 and information on areas calculated and classified by the control device 81 via a communication interface 82, receive operation commands from the outside, and output operation commands to the robot control device 7 based on the received information. The wireless communication device 14 is connected to a remote control device 15 that remotely controls the robot apparatus 1 shown in Figure 12, transmits information output by the robot apparatus 1 to the remote control device 15, and receives operation commands output by the remote control device 15.

[0061] The remote control device 15 is a computer to which a control device 151, a communication interface 152, an input device 153, and a display device 154 are electrically connected. The control device 151 includes a CPU, RAM, ROM, etc., and is configured to execute information processing based on programs and various data. The communication interface 152 is an interface that connects to the wireless communication device 14 and transmits and receives operation commands to the robot device 1 and the status of the robot device 1. The input device 153 is a device that receives operation input from the user using a joystick, keyboard, etc. shown in FIG. 12, and generates operation commands to the robot device 1. The display device 154 is a device that displays the status of the robot device 1 to the user, such as a monitor shown in FIG. 12.

[0062] The robot device 1 autonomously performs tasks as described above. However, if it detects a condition, such as the motor rotation angle of the robot arm 10 not operating according to the command value or excessive motor current, it determines that the task is not being completed and issues a request to the remote control device 15 via the wireless communication device 14 to execute the operation. An operator (not shown) monitors the content of the display device 154 and operates the robot device 1 via the input device 153 based on the request presented by the remote control device 15. At this time, the display device 154 displays a three-dimensional overhead image, as shown in FIG. 13B, to show the overall status of the robot device 1 to the operator, and also displays an image captured by the hand camera 13, as shown in FIG. 13A, to show the detailed status of the work site. The attention area may contain objects that cannot be detected by the environmental sensors. FIG. 13A shows the presence of a valve not included in the environmental map and a broken cable. The operator monitors the screen output of FIG. 13A and operates the robot device 1 to avoid objects not included in the environmental map within the attention area and not detected by the environmental sensors.

[0063] In this way, if the robot device 1 is unable to perform a task properly, when switching to manual operation by the operator, the classified areas can be presented to alert the operator when remotely operating the robot device, allowing the robot device to perform the task properly when remotely operated.

[0064] (Summary) The robot control system 100 in the fifth embodiment further includes a remote operation device 15 that remotely operates the robot device 1, and a wireless communication device 14 that communicates between the robot device 1 and the remote operation device 15. The wireless communication device 14 transmits the state of the robot device 1 and the environmental information that distinguishes between the attention area and an area other than the attention area to the remote operation device 15. The remote operation device 15 displays the state of the robot device 1 and also displays the environmental information that distinguishes between the attention area and an area other than the attention area.

[0065] According to the fifth embodiment configured as described above, the operator remotely controlling the robot device 1 can grasp the state of the robot device 1 and the range of the attention area, and therefore it becomes possible to operate the robot device 1 appropriately by remote control.

[0066] The present invention is not limited to the above-described embodiment, but includes various modifications.

[0067] For example, although the robot device 1 used in the description of this embodiment is illustrated as being configured to move using a wheeled mobile carriage 9, it may be configured to move using a stage or legs. Also, although the robot arm 10 is illustrated as a vertical articulated robot, it may be an orthogonal coordinate robot, a horizontal articulated robot, a parallel link robot, or the like.

[0068] The robot control device 7 is configured to operate upon receiving a trajectory command from the trajectory planning device 8, but may also be configured to receive different operation commands. For example, it may be configured to acquire an operator's operation command generated by lever operation or the like via a network and operate in accordance with that command, or it may be configured to be able to switch the command it acquires.

[0069] The environment setting program 850 reads out pre-stored environment data 841, but the environment data 841 may be configured to be acquired via a network. Also, although SLAM is used as the environment construction method, it is clear that Neural Radiance Fields (NeRF), which uses a model trained to construct a three-dimensional environment model from an image, may also be used, or a different method of constructing an environment model using a different environmental sensor may also be used.

[0070] 1...Robot device, 7...Robot control device, 8...Trajectory planning device, 9...Mobile cart, 10...Robot arm, 11...End effector, 12...Camera (environmental sensor), 13...Hand camera (environmental sensor), 14...Wireless communication device, 15...Remote control device, 21, 22...Inspection work position, 33...Support, 34...Equipment, 71...Control device, 72...Communication interface, 73...Control interface, 74...Storage device, 81...Control device, 82...Communication interface, 83...Input device, 84...Storage device, 10 0...robot control system, 151...control device, 152...communication interface, 153...input device, 154...display device, 741...control program, 841...environmental data, 843...work data, 844...trajectory setting data, 850...environment setting program, 851...spatial recognition unit, 860...trajectory planning program, 861...target position setting unit, 862...operation area setting unit, 863...trajectory planning unit, 870...environmental error determination program, 880...environmental state storage database, S100 to S105...steps.

Claims

1. A robot control device for controlling a robot device equipped with a robot arm having a rotating joint or a parallel moving joint, and an end effector attached to the tip of the robot arm, comprising: a trajectory planning device which sets a target position for the end effector, and plans a motion trajectory of the robot arm based on the target position and an environmental model which is a three-dimensional model of the surrounding environment of the robot device; and an environmental sensor which acquires information of the surrounding environment as environmental information, wherein the trajectory planning device matches the environmental information acquired by the environmental sensor with the environmental model, and sets an area adjacent to the environmental model from which the environmental sensor was unable to acquire environmental information as an attention area, and plans the motion trajectory based on the environmental model, the target position, and the attention area, and the robot control device operates the robot arm based on the motion trajectory.

2. A robot control device according to claim 1, characterized in that the trajectory planning device plans the movement trajectory so that the movement of the robot arm in the attention area is restricted.

3. A robot control device as described in claim 1, characterized in that the trajectory planning device changes the range of the attention area depending on the degree of agreement between the environmental information acquired by the environmental sensor and the environmental model.

4. A robot control device as described in claim 3, characterized in that the trajectory planning device acquires a point cloud of the surrounding environment as the environmental information, calculates the average error between the point cloud and the environmental model as the degree of agreement, and sets a range obtained by expanding the average error by a predetermined amount from the environmental model as the range of the attention area.

5. A robot control device as described in claim 1, characterized in that the trajectory planning device plans the movement trajectory by restricting the position or posture of the robot arm so that the environmental sensor acquires environmental information within the attention area during operation of the robot device.

6. A robot control device as described in claim 1, characterized in that the trajectory planning device stores the difference between the environmental information acquired by the environmental sensor during operation of the robot device and the environmental model as an environmental error, and plans the operation trajectory based on the environmental model, the target position, the attention area, and the environmental error.

7. A robot device comprising: the robot control device according to claim 1; the robot arm; and the end effector.

8. A robot control system comprising: the robot control device according to claim 1; and the robot device.

9. A robot control system as described in claim 8, further comprising a remote operation device for remotely operating the robot device, and a wireless communication device for communicating between the robot device and the remote operation device, wherein the wireless communication device transmits to the remote operation device the state of the robot device and the environmental information distinguishing between the attention area and an area other than the attention area, and the remote operation device displays the state of the robot device and also displays the environmental information distinguishing between the attention area and an area other than the attention area.

10. A robot control method for controlling a robot device comprising a robot arm having a rotating joint or a joint that moves in parallel with the robot arm, and an end effector attached to the tip of the robot arm, comprising: a first step of acquiring environmental information about the surroundings of the robot device; a second step of matching the environmental information acquired in the first step with an environmental model that is a three-dimensional model of the surrounding environment of the robot device, and setting an area adjacent to the environmental model where the environmental information could not be acquired as an attention area; a third step of setting a target position for the end effector; a fourth step of planning a motion trajectory of the robot arm based on the environmental model, the target position, and the attention area; and a fifth step of operating the robot arm based on the motion trajectory.

Images