Monitoring device equipped with a camera for capturing a moving image of a movement of the robot device

Abstract

A monitoring device for monitoring a movement of a robot device (5, 6, 7) comprising a robot (1) and a work tool (2) attached to the robot, comprising: a camera (10) for capturing a moving image of a movement of the robot device; a state detector (18) for detecting an operating state of the robot device; and an arithmetic processing device (4) for obtaining the moving image captured by the camera and an output of the state detector; wherein the arithmetic processing device comprises: a storage unit (42) for storing predetermined information; a memory control unit (32) for handling the information stored in the memory unit; an extraction unit (34) for extracting a part of the information stored in the storage unit; and a determination unit (33) for determining the operating state of the robot device, wherein the memory control unit performs a control to: - storing a moving image in the storage unit, the image being captured by the camera and provided with a time or an elapsed time starting from a predetermined reference time; - storing a variable number in the storage unit, the variable number being obtained from the output of the state detector and provided with the time or the elapsed time; and - Deleting a moving image captured before a predetermined period of time preceding a current time; wherein the determination unit determines whether the operating state of the robot device deviates from a predetermined reference operating state, wherein, if the determination unit determines that the operating state of the robot device deviates from the reference operating state, the extraction unit extracts from the storage unit a moving image acquired in a predetermined time interval including a time interval preceding the deviation from the reference operating state, and stores an extracted moving image in the storage unit, and wherein the memory control unit is configured to perform control for: - Obtaining, as the variable number, at least one of a group of an acceleration in at least one axial direction of a reference coordinate system (56) of the robot, a position of the robot in a reference coordinate system of the robot, a vibration of the robot, a temperature of the robot, a torque output from the robot, a speed of the robot and a force exerted on the robot, - Winning a time at which the variable number is obtained, - providing the variable number with the time and storing it in a monitoring information file (73), and - Recording the operating state of the robot device as monitoring information in the monitoring information file during the operation of the robot device.

Description

BACKGROUND OF THE INVENTIONThe present invention relates to a monitoring device equipped with a camera for capturing a moving image of a movement of the robot device.In a robot device including a robot, a desired operation can be performed by attaching a work tool belonging to the operation to the robot. The work tool may be, for example, a hand attached to the robot to grasp a workpiece. After the robot reaches a predetermined position and posture, a workpiece can be handed over by the hand holding or releasing the workpiece.The robot and the working tool are driven based on a generated motion program. However, the operation of the robot device may not be normally performed depending on the occurrence of an abnormality or a state in the vicinity of the robot device being driven. A camera may be disposed in the robotic device to identify a state of the robotic device or a state in the environment of the robotic device during an operating interval of the robotic device.In the related art, a driving state of a robot is determined by a known device based on a moving image or a still image captured by a camera (for example, JP 2017-013 214 A, JP 2017-001 114 A, and JP 2017-154 221 A). Further, in a known robot device that allows cooperation of an operator and a robot, a camera for recognizing the operator approaching the robot is provided (for example, WO 2018 / 131 237 A1). Further, in a known device, an image of a region of an operating robot device is captured by a camera, and the image is displayed on a display unit (for example, JP 2018-202 541 A).DE 10 2015 104 582 A1 discloses a system and a method for calibrating a robot on a work area, in which at least one camera is arranged on an adjustable component of the robot, at least one robot position is approached with the component in such a way that at least one marker is arranged in an image area of the camera, wherein the image area is recorded with the camera in the robot position, a marker image position of the marker in the recorded image area is determined for each of at least three different recorded image areas. A marker robot position is determined as the position of the marker in the robot coordinate system and, when calibrated after a displacement of the robot or an insertion of another robot and / or later, the steps are carried out again and a deviation is determined from determined marker robot positions with respect to previously determined marker robot positions.EP 1 424 173 A2 discloses an operation state analyzer for a robot for use in analyzing a cause of a failure or an abnormality in an operation of the robot using information of images of the robot in operation and information on an operation state of the robot. Conditions for the start and the end of the information acquisition are set. When the condition for starting information acquisition is satisfied, the image data of the robot acquired by a camera, a state of I / O signals, and information about movement of the robot are recorded every predetermined period until the condition for ending information acquisition is satisfied. The recorded images of the robot are displayed in the form of symbols in a time series. When one of the icons is selected, internal information on the state of the I / O signal, the movement of the robot, etc. associated with the selected icon are displayed. Also, speeds, accelerations, and torques of the respective axes, and a speed, an acceleration, and a torque of a tool center point of the robot are displayed. Since the images and the internal information of the robot acquired during the operation are displayed in time series, analysis of the cause of a failure or an abnormality in the operation of the robot is easy to perform.WO 2018 / 215 047 A1 discloses a method for calibrating a robot coordinate system of a robot with a conveyor coordinate system of a movable conveying element. The method includes: providing a sensor configured to detect positions of the robot in a non-contact manner; detecting a position of the robot when the conveyor member is positioned in a first operating position; detecting a position of the robot and / or the conveyor member by the sensor in the sensor coordinate system when the conveyor member is positioned in a second operating position different from the first operating position; and determining a relationship between the robot coordinate system and the conveyor coordinate system based on at least one detected position of the robot in the sensor coordinate system.US 2018 / 0 243 911 A1 discloses a robot including an instruction receiving unit that receives a calibration initiation instruction, and an arm that changes a positional relationship between a marker indicating a reference point and a detection unit when the calibration initiation instruction is received. Calibration of a coordinate system of the detection unit and a coordinate system of the robot is performed based on an image in which the marker is detected by the detection unit after the positional relationship between the detection unit and the marker is changed.US 2018 / 0 243 898 A1 discloses a robot system comprising a main device configured to receive an instruction for operation from an operator, a sub-arm, a storage device configured to store operation sequence information defining processing performed by the sub-arm, and a control device configured to control the operation of the sub-arm. The control device comprises a receiver configured to receive an input signal, a motion control device configured to determine whether the operation mode of the slave arm is to be automatic, manual or correctable automatic mode and controls the operation of the slave arm in the determined operation mode, and continuation means configured to determine whether continuation of the automatic mode is permitted. In a process in which the slave arm is to be operated in the automatic mode, after the control device interrupts the operation of the slave arm in the automatic mode at a certain process step, the determining means determines whether the continuation of the automatic mode is permitted when the operation is interrupted, based on the input signal received from the receiver.US 2016 / 0 346 932 A1 discloses an automatic calibration method for a robot system. The robot system automatic calibration method includes the steps of calibrating a sensor and a sensor coordinate system of the sensor with respect to a world coordinate system, controlling a robot under the guidance of the sensor to move a point of a tool mounted on the robot to reach a same target point having a plurality of different poses, the point of the tool being located in a tool coordinate system, and calculating a transformation matrix of the tool coordinate system with respect to a tool center point coordinate system based on orientation data of the robot at the same target point.SUMMARY OF THE INVENTIONIf an operation of a robot device is not normally performed, the robot device may be stopped. In some cases, for example, a predetermined part may be attached or welded to a workpiece that is carried over by an automatically guided vehicle. In this case, the workpiece placed on the vehicle may vibrate due to uneven surface of the ground or due to air flow in a factory. As a result, the robot device can detect and stop vibrations when a work tool comes into contact with the workpiece. Alternatively, in the case of malfunction of a device included in the robot device, a robot may largely vibrate and the robot device stops.In order to examine the cause of an abnormality in the robot device, it is preferable to visually observe a state of the robot device when the abnormality occurs. When an operation is not normally performed by the robot device, an operator for visual observation acquires a moving image with a camera while operating the robot device to perform the same operation again. Therefore, the analysis of the cause for the operator is a time-consuming work.The operator analyzes the cause of the abnormality based on the moving images and records of the operations of the robot device. However, a relationship between an operation record of the robot device and a time in a moving image cannot be identified. Since information regarding a relationship between an operation record of the robot device and a time in a moving image is not provided, it is difficult to analyze the cause of an abnormality with respect to the moving image and the operation record of the robot upon occurrence of the abnormality.Therefore, in the related art, it is difficult to analyze the cause of an abnormality in a short time, resulting in a long recovery time for the robot device.According to the present disclosure, there is provided a monitoring apparatus according to the independent claim. Developments are set forth in the dependent claims.Brief Description of the DrawingsFIG. 1 is a schematic diagram of a first robot device according to an embodiment. FIG. 2 is a block diagram of the first robot device according to the embodiment. FIG. 3 illustrates an example of video captured by a camera according to the embodiment. FIG. 4 is a block diagram for explaining objects included in a monitoring information file according to the embodiment. FIG. 5 is a schematic diagram for explaining interpolation points between movement points described in a movement program. FIG. 6 is a schematic diagram of a second robot device according to the embodiment. FIG. 7 is a schematic diagram of a third robot device according to the embodiment.DETAILED DESCRIPTIONWith reference to FIGS. 1 to 7, a monitoring device that monitors a movement of a robot device according to an embodiment will be described below. In the present embodiment, the robot device will be described as an example of assembling parts.FIG. 1 is a schematic diagram of a first robot device according to the present embodiment. A first robot device 5 includes a hand 2 that functions as a working tool (end effector) and a robot 1 that moves the hand 2. The robot 1 of the present embodiment is an articulated robot including a plurality of joints.The robot 1 includes a base part 14 and a rotary base 13 supported by the base part 14. The base part 14 is fixed to an installation surface. The rotating base 13 is configured to rotate relative to the base part 14. The robot 1 includes an upper arm 11 and a lower arm 12, and the lower arm 12 is pivotally supported by the rotary base 13 via a joint. The upper arm 11 is pivotally supported by the lower arm 12 via a hinge. The upper arm 11 rotates about a rotation axis parallel to a direction in which the lower arm 11 extends.The robot 1 includes a wrist 15 connected to one end of the upper arm 11. The wrist 15 is pivotally supported by the upper arm 11 via a joint. The wrist 15 includes a rotatably formed flange 16. the hand 2 is fixed to the flange 16 of the wrist 15. The robot 1 of the present embodiment is a robot having six drive axes, but the embodiment is not limited thereto. Any robot capable of moving the work tool may be used.The hand 2 is a working tool that grips and releases a workpiece 81. The hand 2 has a plurality of claws 3. The hand 2 is configured to open and close the claws 3. The claws 3 grip the workpiece 81 to grip the workpiece 81. The hand 2 of the present embodiment has the claws 3, but the configuration is not limited thereto. The hand may have any configuration configured to grasp the workpiece. For example, a hand that grasps a workpiece by suction or a magnetic force may be used.The robot device 5 of the present embodiment includes a conveyor 8 that functions as a transporter for transferring a workpiece 82 to the robot 1. The transporter is disposed in the vicinity of the robot 1. The conveyor 8 is configured to bring the workpiece 82 to a predetermined position.The robot device 5 of the present embodiment fixes the workpiece 81 to the workpiece 82. the workpiece 81 has a pin 81 aprotruding from the surface. The workpiece 82 to be transferred by the conveyor 8 has a hole 82a conforming to the shape of the pin 81a. The robot device 5 grasps the workpiece 81 disposed on a work table that is not illustrated in the figures. Subsequently, the position and pose of the robot 1 are changed to place the workpiece 81 on the surface of the workpiece 82, as illustrated by an arrow 91. At this point, the robot 1 moves the workpiece 81 to insert the pin 81 ainto the hole 82 a.In the present embodiment, the robot 1 fixes the workpiece 81 to the workpiece 82 while the conveyor 8 advances the workpiece 82. In other words, the workpiece 82 is moved by the conveyor 8 at a time interval in which the workpiece 81 is placed. The robot 1 fixes the workpiece 81 to the workpiece 82 while changing its position and posture to follow the workpiece 82.The robot device 5 of the present invention includes a camera 10 that captures an image of a movement of the robot device 5. The camera 10 is a video camera that captures a moving image. The camera 10 is arranged to capture an image of a state in which the hand 2 performs a work on the workpieces 81 and 82. The camera 10 of the present embodiment is arranged to capture an image of a region where the workpiece 81 is placed on the workpiece 82. Specifically, the camera 10 is arranged to capture an image of a state in which the pin 81 ais inserted into the hole 82 a.In the first robot device 5, the camera 10 is fixed to the hand 2 by a support member 18. The position and orientation of the camera 10 change together with the hand 2. the camera 10 of the present embodiment is configured to be removed from the hand 2. For example, the support member 18 may be fixed to the hand 2 by a magnet or a fastener such as a screw.In the robot device 5 of the present embodiment, a reference coordinate system 56 is defined. In the example of FIG. 1, the origin of the reference coordinate system 56 is located in the base part 14 of the robot 1. the reference coordinate system 56 is also referred to as a world frame (world frame). The reference coordinate system 56 is a reference system in which the position of the origin is fixed and in which the directions of the coordinate axes are fixed. Even if the position and the pose of the robot 1 are changed, the position and the orientation of the reference coordinate system 56 remain unchanged. The reference coordinate system 56 has, as coordinate axes, the X axis, the Y axis, and the Z axis that are perpendicular to each other. Further, the Wachse is defined as a coordinate axis around the X axis. The P axis is defined as a coordinate axis around the Y axis. And the R axis is defined as a coordinate axis around the Z axis.The positions and poses of the robot 1 can be expressed by the reference coordinate system 56. For example, the position of the robot 1 may be expressed by the position of a tool center point disposed at the tip of the hand 2. Furthermore, a tool coordinate system can be defined at the tool center point, which moves with the hand 2. The poses of the robot 1 can be expressed by the orientations of the tool coordinate system relative to the reference coordinate system 56.FIG. 2 is a block diagram illustrating the robot device according to the present embodiment. Referring to FIGS. 1 and 2, the robot 1 includes a robot driving device that changes the position and pose of the robot 1. The robot driving device includes a robot driving motor 22 that drives components such as an arm and a wrist. Driving the robot drive motor 22 changes the orientations of the components.The hand 2 includes a hand driving device that drives the hand 2. The hand driving device includes a hand driving motor 21 that drives the claws of the hand 2. Driving the hand drive motor 21 opens and closes the claws 3 of the hand 2. In this case, the manual driving device may include an air pump or a cylinder, i.e., a device for driving the claws by air pressure.A control unit for the robot device 5 includes a robot control unit 4. the robot control unit 4 includes an arithmetic processing device (computer) having a central processing unit (CPU) as a processor. The arithmetic processing device includes RAM (Random Access Memory) and ROM (Read Only Memory) connected to the CPU via a bus. A motion program 41 generated in advance for controlling the robot 1, the hand 2, and the conveyor 8 is input to the robot control unit 4. The robot 1 and the hand 2 transfer the workpiece 81 based on the motion program 41. the conveyor 8 transfers the workpiece 82 based on the motion program 41.The arithmetic processing device of the robot control unit 4 includes a storage unit 42 that stores predetermined information. The storage unit 42 stores information regarding the control of the robot 1, the hand 2, and the conveyor 8. The storage unit 42 may include a storage medium capable of storing information such as a volatile memory, a nonvolatile memory, or a hard disk.The arithmetic processing device includes a motion control unit 43 that transmits a motion command. The motion control unit 43 is equivalent to a processor operated according to the motion program 41. The motion control unit 43 is configured to read information stored in the storage unit 42. The processor reads the motion program 41 so that the processor functions as the motion control unit 43.The motion control unit 43 transmits a motion command for driving the robot 1 to a robot drive unit 45 based on the motion program. the robot drive unit 45 includes an electric circuit that drives the robot drive motors 22. The robot drive unit 45 supplies electricity to the robot drive motors 22 based on the movement command. The motion control unit 43 transmits the motion command for driving the hand 2 to a hand drive unit 44 based on the motion program 41. the hand drive unit 44 includes an electric circuit that drives the hand drive motor 21. The hand drive unit 44 supplies electricity to the hand drive motor 21 based on a movement command. The motion control unit 43 transmits a command for capturing a moving image to the camera 10 based on the motion program 41.The robot control unit 4 includes a display 46 that displays arbitrary information regarding the robot device 5. The display 46 includes, for example, a liquid crystal display panel.The robot device 5 includes at least one state detector that detects the operating state of the robot device 5. The state detector is fixed to a member constituting the robot device 5. The robot 1 includes a position detector 18 serving as a state detector for detecting the position and pose of the robot 1. The position detector 18 is fixed to the robot drive motor 22 belonging to the drive axis of a component such as an arm. For example, the position detector 18 detects a rotation angle when the robot drive motor 22 is driven. The position and pose of the robot 1 are detected based on the position detector 18.The state detector of the present embodiment includes an acceleration sensor 19 for detecting the acceleration of the robot 1. According to the output of the acceleration sensor 19, the acceleration and vibrations of the robot 1 when driven can be detected.The control unit of the robot device 5 includes a conveyor control unit 9 that controls the operation of the conveyor 8. The conveyor control unit 9 includes an arithmetic processing device (computer) including a CPU and a processor and RAM. The conveyor controller 9 is configured to communicate with the robot control unit 4. The motion control unit 43 transmits a motion command for driving the conveyor 8 to the conveyor control unit 9 based on the motion program 41. the conveyor control unit 9 receives the motion command from the robot control unit 4 and drives the conveyor 8.The robot device 5 of the present embodiment includes a monitoring device that monitors a movement of the robot device 5. The monitoring device includes the camera 10, the position detector 18 serving as a state detector, and the acceleration sensor 19. In the present embodiment, the arithmetic processing device of the robot control unit 4 serves as an arithmetic processing device of the monitoring device.The arithmetic processing device of the robot control unit 4 includes a signal processing unit 31 that processes information output from the state detector and the camera 10. The signal processing unit 31 includes a storage control unit 32 that handles the information stored in the storage unit 42. The signal processing unit 31 includes an extraction unit 43 that extracts a part of the information stored in the storage unit 42. The signal processing unit 31 further includes a determination unit 33 that determines the operation state of the robot device 5. Further, the signal processing unit 31 of the present embodiment includes a prediction unit 35 that predicts future movement of the robot device 5 based on a past operation state of the robot device 5.The signal processing unit 31 is equivalent to the processor operated according to the motion program 41. Specifically, the storage control unit 32, the determination unit 33, the extraction unit 34, and the prediction unit 35 are equivalent to the processor operated according to the motion program 41. The processor reads the motion program 41 and performs control as determined in the motion program 41 so that the processor functions as each of the units.The control unit of the robot device 5 according to the present embodiment includes the robot control unit 4 that controls the robot 1 and the hand 2, and the conveyor control unit 9 that controls the conveyor 8. The embodiment is not limited to this configuration. The robot device 5 may be configured to control the robot 1, the hand 2, and the conveyor 8 by a single control unit. In addition to the robot control unit 4, a signal processing device having the function of the signal processing unit 31 may be provided. The signal processing device may be constituted by a storage unit and an arithmetic processing device (computer) including a CPU. The signal processing device may perform control for storing information in the control unit and control for extracting a part of the information stored in the storage unit. Alternatively, the processing may be shared for signals of the detector, the sensor, and the camera while synchronizing a time in the signal processing device and a time in the robot control unit 4 with each other.Referring to FIG. 1, the camera 10 of the present embodiment is disposed at a position away from the workpiece 81 gripped by the hand 2. The camera 10 is arranged to capture an image of the pin 81 aof the workpiece 81 and the hole 82 aof the workpiece 82 when the workpiece 81 is attached to the workpiece 82.FIG. 3 illustrates an example of a video stored in the storage unit. Referring to FIGS. 2 and 3, the storage control unit 32 acquires a moving image captured by the camera 10. The robot control unit 4 has a function of a clock. The storage control unit 32 acquires the current time. The storage control unit 32 provides a time for a moving image captured by the camera 10 and stores the moving image in the storage unit 42.A state in which the pin 81 aof the workpiece 81 and the hole 82 aof the workpiece 82 transferred from the conveyor 8 are engaged with each other is recorded in a video 71. The workpiece 81 gripped by the claws 3 moves in the direction of the arrow 92, whereby the pin 81a is inserted into the hole 82a. The camera 10 acquires images of a state in which the robot device 5 performs an operation on the workpieces 81 and 82. A time stamp 72 is associated with a predetermined position of the video 71. The video 72 indicates a date and time when the video has been recorded. In the example of FIG. 3, a time is displayed in steps of hundreds of seconds.The storage control unit 32 of the present embodiment executes control for deleting an old moving image stored in the past. The storage control unit 32 performs control for deleting a moving image acquired before a predetermined time interval preceding a current time from the storage unit 42 from the storage unit 42. The storage unit 42 must have a very large storage capacity of moving images. In the present embodiment, an increase in the storage capacity of moving images can be suppressed by deleting old moving images. For example, the storage control unit 32 may perform control for storing, in the storage unit 42, a moving image captured by the camera 10 at a current time and erasing a moving image captured before the time interval preceding the current time by thirty minutes.FIG. 4 is an exemplary diagram of a monitoring information file according to the present embodiment. The storage control unit 32 of the present embodiment performs control for recording an operation state of the robot device 5 as monitoring information in a monitoring information file 73 during an operation time interval of the robot device 5. The monitoring information file 73 is stored in the storage unit 42. The storage control unit 32 performs control for providing a variable number obtained from the output of the state detector with time and storing the variable number in the storage unit 42. The operation state of the robot 1 is, for example, the acceleration of the robot 1 and the position of the robot 1. at least one of the acceleration of the robot 1 and the position of the robot 1 can be recorded in the monitoring information file 73.In the present embodiment, the acceleration of the robot 1 is an acceleration obtained from the output of the acceleration sensor 18. The storage control unit 32 acquires an acceleration in the X-axis direction, an acceleration in the Y-axis direction, and an acceleration in the Z-axis direction of the reference coordinate system 56, the respective accelerations being output from the acceleration sensor 19. The accelerations are timed and stored in the monitoring information file 73. Further, the storage control unit 32 acquires the position of the robot 1, and the position is calculated based on the output of the position detector 18 attached to the robot drive motor 22. The X-axis coordinate value, the Y-axis coordinate value, and the Z-axis coordinate value in the reference coordinate system 56 of the robot are stored in the monitoring information file 73.In the example of FIG. 4, the acceleration of the robot 1 and the position of the robot 1 at the time t 1 are indicated. The monitoring information file 73 stores variable numbers obtained from the output of the state detector at predetermined time intervals. In other words, variable numbers are stored in the monitoring information file 73 at respective times. An actually measured value obtained from the state detector is time stamped and stored in the monitoring information file 73. The variable numbers may be stored in the monitoring information file 73 as time series.In the present embodiment, the storage control unit 32 stores acceleration and position of the robot in the storage unit 42 at each time during a time interval when the robot device 5 is driven. The embodiment is not limited to this configuration. The storage control unit may perform control for deleting a variable number obtained before the predetermined time interval preceding a current time.Referring to FIG. 2, the determination unit 33 determines whether the operation state of the robot device 5 deviates from a predetermined reference operation state during the operation time interval of the robot device 5. The reference operation state corresponds to a state in which the robot device 5 operates within a normal operation range. The determination unit 33 of the first robot device 5 determines whether or not the robot 1, the hand 2, and the conveyor 8 are operating normally. The determination unit 33 determines whether a variable number obtained from the output of the state detector deviates from a predetermined determination range. If a variable number obtained from the output of the state detector deviates from the determination range, it can be determined that the operation state of the robot device 5 has deviated from the reference operation state. In other words, it can be determined that the operation state of the robot device 5 is abnormal.For example, the determination range of the acceleration of the robot may be determined in advance. If an acceleration acquired from the acceleration sensor 19 exceeds the determination range, the determination unit 33 may determine that the operation state of the robot device 5 has deviated from the reference operation state. Alternatively, the determination range of positions that can be reached by the robot 1 may be determined in advance. If a position of the robot device 1 calculated from the output of the state detector 18 deviates from the determination range, the determination unit 33 may determine that the operation state of the robot device 5 has deviated from the reference operation state.A variable number for determining whether the operation state of the robot device 5 deviates from the reference operation state may be identical to or different from variable numbers recorded in the monitoring information file 73. The determination unit may perform any control for determining whether the operation state of the robot device has deviated from the reference operation state. For example, if a command value transmitted from the motion control unit 43 and a measured value, for example, the output of the position detector 18 are different from each other, the determination unit may determine that the operation state of the robot device 5 has deviated from the reference operation state.If the determination unit 33 determines that the operation state of the robot device 5 has deviated from the reference operation state, the signal processing unit 31 displays the deviation of the operation state of the robot device 5 from the reference operation state on the display 46. An operator can be informed of the occurrence of abnormality of the robot device 5. Further, the signal processing unit 31 may transmit a command to stop the robot device 5 to the motion control unit 43. The motion control unit 43 may stop the robot device 5 based on the command. Further, the motion control unit 43 may retract the robot 1 to bring the position of the robot 1 to a safe position based on the command.The extraction unit 34 acquires the time of deviation from the reference operation state. Subsequently, the extraction unit 34 extracts, from the storage unit 42, a moving image at a predetermined time interval including a time interval that preceded the deviation from the reference operation state. The extraction unit 34 extracts a moving image including a moving image that preceded the occurrence of abnormality of the robot device 5. The predetermined time interval that preceded the occurrence of the abnormality of the robot device 5 may be, for example, a time interval of one to ten minutes since the deviation from the reference operation state. In the present embodiment, the extraction unit 34 extracts a moving image at a predetermined time interval including the deviation from the reference operation state. Specifically, the extraction unit 34 of the present embodiment extracts a moving image from the storage unit 42 such that the moving image includes a moving image at a predetermined time interval following the deviation from the reference operation state. The predetermined time interval following the occurrence of an abnormality may be, for example, a time interval of one to ten minutes since the deviation from the reference operation state. In this manner, the extraction unit 34 extracts, from the storage unit 42, a moving image at a time interval around the time point at which the operation state of the robot device 5 deviates from the reference operation state.The extraction unit 34 may not extract a moving image after the deviation from the reference operation state. However, extraction of a moving image after the deviation from the reference operation state allows an operator to confirm a status by a video after occurrence of an abnormality. For example, the operator may confirm a status of damage to the device.The extraction unit 34 then stores the extracted moving image in the storage unit 42. Further, the extraction unit 34 may extract, from the monitoring information file 73, information at a predetermined time interval of occurrence of an abnormality in the operating state of the robot device 5, and store the information in the storage unit 42.In this way, the monitoring device of the present embodiment acquires a moving image of the assembly area of the workpieces 81 and 82 during the operation time interval of the robot device 5.A moving image immediately before the deviation of the robot device 5 from the reference operation state and a variable number obtained from the output of the state detector are stored in the storage unit 42. Therefore, it is not necessary for an operator to operate the robot device 5 again to repeat the same operation state, resulting in rapid analysis regarding the state of the robot device 5 in the event of abnormality.The operator can confirm a state of parts manufacturing by the robot device 5 from the video 71 extracted by the extraction unit 34. Further, the operator can confirm a state of the robot 1 in the case of an abnormality based on an acceleration and the position of the robot, the acceleration and the position being recorded in the monitoring information file 73. Variable numbers included in the video 71 and the monitoring information file 73 are provided with times, resulting in a unique relationship between a time included in the video 71 and a time stored in the monitoring information file 73. Therefore, the operator can easily analyze the cause of an abnormality. For example, the operator acquires the time of deviation of acceleration from the determination area from the monitoring information file 73, thereby confirming a state immediately before the time of deviation of acceleration from the determination area in the video 71. Since a video and an operation state can be confirmed immediately after occurrence of an abnormality in the robot device 5, the cause of the abnormality can be analyzed in a short time.In the present embodiment, the video 71 captured by the camera 10 and a variable number obtained from the output of the state detector are stored with allocated times. The embodiment is not limited to this configuration. Instead of the time, an elapsed time since a predetermined reference time may be attached to a video and a variable number obtained from an output of the state detector. For example, the stored video and the variable number may be provided with the reference timing. A time or an elapsed time to provide the video 71 and the variable number stored in the monitoring information file 73 may be recorded in units of time specific to the device. For example, as shown in FIG. 3, the time may be one hundredths of a second or the elapsed time may be given in units of milliseconds.The camera 10 of the first robot device 5 is configured to be removed from the hand 2. Due to this configuration, the camera 10 can be attached to or detached from the robot device 5 at any time interval. For example, the camera may be removed at a time interval in which the robot device causes normal movement. The camera may be mounted if the robot device is likely to cause abnormal movement. For example, the camera may be mounted to monitor the motion state of the robot device when a warning regarding a motion of the robot device is issued. The camera 10 of the first robot device 5 is attached to the hand 2, but the embodiment is not limited thereto. The camera may be attached to the robot. For example, the camera may be attached to the flange of the wrist of the robot.Time intervals for storing variable numbers obtained from the output of the state detector by the storage control unit 32 in the storage unit 42 may be the same time intervals as a control time interval for transmitting the motion command of the robot 1 by the motion command unit 43. the storage control unit 32 may obtain the output of the state detector simultaneously with a time interval for transmitting the motion command of the robot 1 by the motion control unit 43. Information regarding the variable numbers of times is added to the monitoring information file 73 every control time interval.FIG. 5 is an explanatory diagram of the control time interval transmitted from the motion control unit of the control unit. FIG. 5 indicates moving points P 1, P 2, and P 3 described in the moving program 41. The robot 1 is controlled to move a tool center from the movement point P 1 to the movement point P 2, for example. In the motion program 41, the positions of the motion points P 1, P 2, and P 3 and the poses of the robot 1 are set at the respective positions.The motion control unit 43 transmits the motion command to the robot at a predetermined control time interval Δt. The control time interval Δt is determined in advance according to the performance of the robot 1. Alternatively, the control time interval Δt may be input to the robot control unit 4 by an operator. The control time interval Δt of the robot 1 is, for example, a time that may be from 1 ms to 10 ms.The motion control unit 43 acquires the positions and poses of the robot 1 at the motion points P 1, P 2, and P 3 from the motion program 41. the motion control unit 43 then adds interpolation points IP between the motion points P 1, P 2, and P 3. The interpolation points IP are generated based on the time intervals of the control time interval Δt of the robot 1. The interpolation points IP are also generated based on the driving speed of the robot 1 and the driving method (motion plan) of the robot 1, such as linear motion.The motion control unit 43 calculates the position and pose of the robot 1 at each of the interpolation points IP. Further, the motion control unit 43 calculates the operation speed of the robot 1 at each of the interpolation points IP. The motion control unit 43 transmits a motion command for each of the interpolation points IP. The positions and postures of the robot 1 are controlled to pass through the moving points P 1, P 2, and P 3 and the interpolation points IP. Alternatively, the robot 1 is controlled to move past the moving points P 1, P 2, and P 3 and the interpolation points IP.The storage control unit 32 may store, in the storage unit 42, variable numbers (measured values) obtained from the output of the state detector at each control time interval of the robot 1. Further, the storage control unit 32 may store, in the storage unit 42, a command value transmitted from the motion control unit 43 at each control time interval of the robot 1. The control time interval is a minimum time interval for driving the robot 1. therefore, lack of records regarding the state of the robot can be suppressed by recording variable numbers in the monitoring information file 73 at the time intervals of the control time intervals. In addition, an operator can analyze the abnormality of movement of the robot device 5 at the interpolation points IP.Any monitoring information for monitoring the operation state of the robot device 5 may be stored in the monitoring information file 73 with an associated time. The monitoring information may include, for example, warning information that does not require stopping of the robot device 5. For example, in addition to an acceptance value for determining the abnormality of movement of the robot 1 with respect to acceleration, a determination range for a warning regarding approach to the acceptance value may be determined in advance. If the determination unit 33 recognizes that the acceleration of the robot 1 has reached the determination range of a warning, the storage control unit 32 may provide information regarding the warning with time and store the information in the monitoring information file 73.The monitoring information may include information regarding the command expression of the motion program 41 corresponding to a motion of the robot device 5. For example, information regarding the text lines of the command expression for executing the motion program 41 may be recorded together with times. Alternatively, the information regarding the lines of the command expression for executing the motion program 41 may be recorded together with variable numbers such as acceleration.In the motion program 41, motion groups for devices constituting the robot device 5 may be defined. A motion group for driving the robot 1, a motion group for driving the hand 2, and a motion group for driving the conveyor 8 may be determined for the motion program 41. The monitoring information may include information regarding the motion groups. For example, a first motion group for driving the robot 1 may be recorded together with variable numbers such as acceleration in the monitoring information file 73.In the monitoring information, information regarding the information points between the movement points described in the movement program may be included together with times. For example, the monitoring information may include information regarding the control time interval between the interpolation points and the motion plan. The monitoring information may also include information regarding positions, speeds, accelerations, and vibrations at the interpolation points. In this case, the positions, speeds, and the like at the interpolation points can be calculated by obtaining the output of the state detector by the storage control unit 32 at the same time intervals as the control time interval. Further, the information regarding the interpolation points may be the command value of the position of the robot 1, the command value being transmitted from the motion control unit 43. Alternatively, the monitoring information may include both a measured value calculated from the output of the state detector and a command value transmitted from the motion control unit 43.At least one of the warning information, the information regarding the command expression of the motion program, and the information regarding the interpolation points is recorded in the monitoring information file 73, which allows an operator to more accurately analyze the cause of deviation of the robot device 5 from the reference operation state.Referring to FIG. 2, the signal processing unit 31 of the present embodiment includes the prediction unit 35 predicting future movement of the robot 1. The prediction unit 35 acquires the monitoring information file 73 of the previously recorded operating states of the robot from the storage unit 42. The prediction unit 35 estimates the position of the robot 1 after the robot 1 is driven by a subsequent movement command output at each control time interval, based on variable numbers detected from the output of the state detector. For example, the prediction unit 35 may estimate the position of the robot after the robot will be driven by a subsequent movement command from the current position and the previous position of the robot according to a linear extrapolation. The prediction unit 35 repeats control for estimating a position at which the robot will have arrived by a subsequent movement of the robot for each control time interval.The determination unit 33 determines whether the robot 1 can reach the position of the robot 1 estimated by the prediction unit 35. For example, the estimated position of the robot 1 may exceed the range of movement of the robot 1. Further, the rotation angle of the robot drive motor 22 at each drive axis is calculated based on inverse kinematics when the position of the robot is determined. At this point, the rotation angle of the robot drive motor 22 may not be calculated. This moving point of the robot 1 is referred to as a singular point. In this case, the determination unit 33 determines that the position of the robot 1 after the robot will be driven by a subsequent movement command is a position that cannot be reached by the robot 1.If the robot 1 cannot reach the position estimated by the prediction unit 35, the signal processing unit 31 transmits a command to stop the robot 1 to the motion control unit 43 before the robot 1 is driven by a subsequent motion command. The motion control unit 43 performs control for stopping the robot 1 without transmitting a subsequent motion command based on the command from the signal processing unit 31.If the position and pose of the robot are corrected by using a visual sensor, a plurality of positions can be reached by the robot 1 according to a subsequent movement command. In this case, the determination unit 33 may determine whether each position exceeds the range of movement or is a singular point. If at least one position exceeds the range of movement or is a singular point, the determination unit 33 may determine that the estimated position of the robot 1 cannot be reached by the robot 1.In the robot device 5, visual feedback control for correcting the position and pose of the robot 1 may be corrected based on a moving image or a still image of a workpiece moved by a conveyor or the like, the image being captured by a visual sensor such as a camera. In the visual feedback control, the position and posture of the robot 1 are changed to follow a workpiece moved by a conveyor or the like.For example, the position and the pose of the robot 1 may be adjusted such that the characteristic point of a moving workpiece in a video captured by a camera is at a predetermined reference position. The positional relationship of the robot relative to the workpiece is determined in advance when the characteristic point of the workpiece is at the reference position in the video. Therefore, the robot controller 4 can control the robot 1 to control the position and the pose of the robot 1 to perform an operation based on the position and the pose of the robot 1 when the characteristic point of the workpiece is at the reference position in the video.Further, the position and orientation of the workpiece in the reference coordinate system 56 may be calculated based on the video captured by the camera when the workpiece is moved by the conveyor or the like. For example, the position and orientation of the workpiece can be calculated according to a pattern recognition method. The robot control unit 4 may control the robot 1 such that the robot 1 is at the position and in the pose for performing a work, based on the position and the pose of the workpiece at the time when an image is captured by the camera.In the visual feedback control, the position and pose of the robot 1 are changed according to the position and orientation of the workpiece, which leads to difficulties in estimating a future position and pose of the robot in advance. However, the prediction unit 35 of the present embodiment may estimate the position of the robot at a time close to the current time based on past monitoring information. Therefore, the robot 1 can be stopped before the movement of the robot 1 becomes abnormal. Specifically, variable numbers obtained from the output of the state detector at each time interval are stored in the storage unit 42 by the storage control unit 32, whereby the position of the robot is estimated after the robot will be driven by a subsequent movement command.In the present embodiment, the acceleration sensor 19 is attached to the hand 2 to detect the acceleration of the robot 1. The embodiment is not limited thereto. For example, camera 10 may include an accelerometer to compensate for image blurring caused by hand-held camera shake. If the camera 10 includes an acceleration sensor, the output value of the acceleration sensor of the camera may be used as the acceleration of the robot 1. Alternatively, the acceleration of the robot 1 cannot be directly detected by the acceleration sensor. The acceleration can be calculated by deriving twice the output of the position detector 18. For example, an acceleration at a predetermined position, for example, at a tool center point, may be calculated based on the output of the position detector 18.FIG. 6 is a schematic diagram of a second robot device according to the present embodiment. In the first robot device 5, the camera 10 is attached to the hand 2. The camera 10 travels with the driven robot 1, but the embodiment is not limited thereto. The camera 10 may be fixed at a position away from the robot 1 and the conveyor 1.A second robot device 6 comprises a base 25 which is fastened to a base. The camera 10 is fixed to the base 25. The camera 10 is arranged to capture an image of a state in which the hand performs a work operation on the workpieces 81 and 82. The camera 10 is arranged to capture an image of a state in which the pin 81 aof the workpiece 81 is inserted into the hole 82 aof the workpiece 82.Further, the second robot device 6 can acquire an image of an area different from the area where the robot device 5 performs an operation. By adopting this configuration, an operator can confirm in the video that a phenomenon occurs in a region different from the region in which the work tool performs the work operation on the workpiece. For example, the camera 10 may be arranged to capture an image of the entire robot 1. By adopting this configuration, the camera 10 can capture the images of all the movements of the robot 1. For example, an operator in the video can confirm that an operator or another device comes into contact with the rotary base 13 of the robot 1. Other configurations, operations, and effects of the second robot device 6 are identical to those of the first robot device 5.FIG. 7 is a schematic diagram of a third robot device according to the present embodiment. In the first robot device 5 and the second robot device 6, the workpiece 82 is transferred by the conveyor 8. The embodiment is not limited thereto. The third robot device 7 includes a transport vehicle 26 serving as a transporter for transferring the workpiece 82 to the robot 1. The transport vehicle 26 comprises wheels 28 and travels in any direction on the ground. The transport vehicle 26 of the present embodiment is a keyless vehicle that automatically transfers the workpiece 82 based on the motion program 41.The third robot device 7 includes a transport vehicle control unit that controls the transport vehicle 26. The transport vehicle control unit includes an arithmetic processing unit (computer) including a CPU as a processor and RAM. The transport vehicle control unit is configured to communicate with the robot control unit 4.The workpiece 82 is fixed on an upper surface of the transport vehicle 26 with a fastener 27 interposed therebetween. The transport vehicle 26 moves along a predetermined path. The position and posture of the robot 1 are changed to insert the pin 81 aof the workpiece 81 into the hole 82 aof the workpiece 82. In the third robot device 7, the camera 10 is fixed to the transport vehicle 26 with a support member 29 interposed therebetween. The camera 10 moves with the transport vehicle 26.In addition, in the third robot device 7, the same control as in the first robot device 5 can be performed. Also in the case where the transport vehicle 26 is used as a transporter, the camera 10 can be mounted at a position remote from the transport vehicle 26 as in the second robot device 6 (see FIG. 6 ). Other configurations, operations, and effects of the third robot device 7 are identical to those of the first robot device 5 and the second robot device 6.The robot control unit 4 of the present embodiment includes the display 46 that displays the monitoring information file 73 and the video 71. The embodiment is not limited thereto. The robot control unit may not include a display. The monitoring information file and the video extracted by the extraction unit may be transmitted to another device via a communication device. For example, information may be transmitted to an operator in the robot device abnormality analyzing section.The embodiment has illustrated the robot device that performs a work operation on the workpiece 82 moved by the transporter, but the embodiment is not limited thereto. The monitoring device of the present embodiment can also be applied to a robot device that performs a work on a stopped workpiece.The embodiment has shown the position detector and the acceleration sensor as state detectors. The embodiment is not limited thereto. The state detector may be any detector capable of detecting a state of the robot device. For example, the state detector may include a temperature sensor, a torque sensor, a speed sensor, or a force sensor attached to the robot device.The embodiment illustrates the robot device for assembling parts. The embodiment is not limited thereto. The monitoring device of the present embodiment can be applied to a robot device for performing any operation. For example, the monitoring device of the present embodiment can be applied to a robot device for transferring a workpiece, a robot device for welding such as spot welding or arc welding, a robot device for painting, a robot device for applying a sealant, or the like.According to the present disclosure, the monitoring device is provided to be able to easily confirm a state of movement of the robot device when the movement of the robot device deviates from a reference operation state.In the described control, the order of the steps may be optionally changed as long as the functions and effects are not changed.

Claims

A monitoring device for monitoring a motion of a robot device (5, 6, 7) including a robot (1) and a work tool (2) attached to the robot, comprising: a camera (10) for acquiring a moving image of a motion of the robot device; a state detector (18) for detecting an operation state of the robot device; and an arithmetic processing device (4) for acquiring the moving image acquired by the camera and an output of the state detector; wherein the arithmetic processing device includes: a storage unit (42) for storing predetermined information; a storage control unit (32) for managing the information stored in the storage unit; an extraction unit (34) for extracting a part of the information stored in the storage unit; A determination unit (33) for determining the operation state of the robot device, wherein the storage control unit performs control for: storing a moving image in the storage unit, the image being captured by the camera and provided with a time or an elapsed time from a predetermined reference time point; storing a variable number in the storage unit, the variable number being obtained from the output of the state detector and provided with the time or the elapsed time point; and deleting a moving image captured before a predetermined time period that preceded a current time point; wherein the determination unit determines whether the operation state of the robot device deviates from a predetermined reference operation state, wherein if the determination unit determines that the operation state of the robot device deviates from the reference operation state, the extraction unit extracts a moving image from the storage unit, which has been acquired in a predetermined time interval including a time interval preceding the deviation from the reference operation state and stores an extracted moving image in the storage unit, and wherein the storage control unit is configured to perform control for: acquiring, as the variable number, at least one of a group of acceleration in at least one axis direction of a reference coordinate system (56) of the robot, a position of the robot in a reference coordinate system of the robot, vibration of the robot, a temperature of the robot, torque output from the robot, a speed of the robot, and a force applied to the robot, acquiring a time at which the variable number is acquired, providing the variable number with the time, and storing it in a monitoring information file (73), recording the operating state of the robot device as monitoring information in the monitoring information file during the operation of the robot device.The monitoring device according to claim 1, wherein the camera is arranged to capture an image of a state in which the work tool performs a work operation on the workpiece (81, 82).The monitoring device according to claim 1 or 2, wherein if the determination unit determines that the operation state of the robot device deviates from the reference operation state, the extraction unit extracts a moving image at a time interval following the deviation from the reference operation state, and stores an extracted moving image in the storage unit.The monitoring device according to any one of claims 1 to 3, wherein the arithmetic processing device includes a motion control unit (43) for transmitting a motion command to the robot at each predetermined control interval, and wherein the storage control unit stores, in the storage unit, variable numbers obtained from the output of the state detector at the same time intervals as a control interval of the robot.The monitoring device according to any one of claims 1 to 4, wherein the storage control unit includes, in the storage unit, at least one of information regarding a warning of an operation of the robot device, information regarding a control instruction of a motion program (41) corresponding to a motion of the robot device, and information regarding interpolation points (IP) generated between motion points (P1, P2, P3) described in the motion program, the information being provided with a time or the elapsed time.The monitoring device according to claim 4, wherein the arithmetic processing device includes a prediction unit (35) for predicting a motion of the robot, the prediction unit estimating a position of the robot after the robot is driven according to a subsequent motion command, based on the variable number obtained from the output of the state detector and stored in the storage unit, wherein the determination unit determines whether the robot position estimated by the prediction unit can be reached by the robot, and if the robot position estimated by the prediction unit cannot be reached by the robot, the robot is stopped before being driven according to the subsequent motion command.

Images