Teaching data correction device

Abstract

To solve the problem that teaching data cannot be corrected during operation of a robot.SOLUTION: A teaching data correcting device 3 comprises: a model storage part 31 that stores a three-dimensional model of a virtual robot corresponding to an actual robot in an actual environment; a teaching data storage part 32 that stores teaching data on the actual robot; a receiving part 34 that receives operation and teaching instructions to the three-dimensional model; a positional relation obtaining part 33 that obtains a relative positional relation between a display device 4 that displays an image while superimposing the image on an image of the actual environment or on the actual environment and the actual environment; an image generating part 35 that generates a display image for displaying the operated three-dimensional model so that the three-dimensional model is in a predetermined relation with the actual robot, using the relative positional relation with the three-dimensional model; an output part 36 that outputs the display image to the display device 4; and a correcting part 39 that corrects the teaching data in accordance with the three-dimensional model, when receiving the teaching instruction during operation of the actual robot.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a teaching data correction device for correcting teaching data. [Background technology]

[0002] Conventionally, robots have been controlled by a teaching playback method, in which teaching is usually performed by operating the robot and teaching data is created. Summary of the Invention [Problem to be solved by the invention]

[0003] In addition, when the teaching data is not appropriate or when there is a change in the workpiece, etc., it may be desirable to modify the teaching data after teaching. However, there is a problem in that the teaching data cannot be modified using the robot while the robot is in operation. This is because stopping the robot to modify the teaching data reduces productivity. Also, for example, in the case of a welding robot, there is a problem in that stopping the operation in the middle of welding can result in a defective product being welded.

[0004] The present invention has been made to solve the above-mentioned problems, and has an object to provide a teaching data correction device that can correct teaching data while a robot is in operation. [Means for solving the problem]

[0005] In order to achieve the above-mentioned object, a teaching data correction device according to one embodiment of the present invention includes a model memory unit in which a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment is stored, a teaching data memory unit in which teaching data of the real robot is stored, a reception unit for receiving operation and teaching instructions for the three-dimensional model of the virtual robot, a positional relationship acquisition unit for acquiring a relative positional relationship between the real environment in which the real robot exists and a display device that displays an image superimposed on an image of the real environment or the real environment itself, an image generation unit for generating a display image for displaying the operated three-dimensional model so that it is in a predetermined positional relationship with the real robot based on the relative positional relationship with the three-dimensional model, an output unit for outputting the display image to the display device, and a correction unit for correcting the teaching data in accordance with the three-dimensional model of the virtual robot when a teaching instruction for the three-dimensional model of the virtual robot is accepted by the reception unit while the real robot is operating. Effect of the Invention

[0006] According to one embodiment of the teaching data correction device of the present invention, the teaching data can be corrected by using a three-dimensional model of a virtual robot even while the real robot is operating, making it possible to correct the teaching data without reducing productivity. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing a robot control system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a configuration of a teaching data correction device according to the embodiment; [Diagram 3] A flowchart showing the operation of the teaching data correcting device according to the embodiment. [Figure 4] A flowchart showing the operation of the teaching data correcting device according to the embodiment. [Figure 5A] FIG. 13 is a diagram showing an example of a period for which teaching data is to be corrected in the embodiment; [Figure 5B] FIG. 13 is a diagram showing an example of an area to be corrected in the teaching data according to the embodiment; [Figure 6] FIG. 13 is a diagram showing an example of correction of teaching data in the embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The teaching data correction device according to the present invention will be described below using an embodiment. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated explanation may be omitted. According to the teaching data correction device according to this embodiment, the teaching data is corrected using a 3D model of a virtual robot corresponding to a real robot, so that the teaching data can be corrected even while the real robot is in operation.

[0009] Fig. 1 is a schematic diagram showing the configuration of a robot control system 100 according to this embodiment. As shown in Fig. 1, the robot control system 100 includes a robot 1, a robot control device 2, a teaching data correction device 3, and a display device 4. Note that the devices may be connected to each other, for example, by wire or wirelessly.

[0010] The robot 1 is usually an industrial robot, and may be a manipulator having a plurality of arms (links) connected by joints driven by motors. The robot 1 may be, for example, a vertical articulated robot or a horizontal articulated robot. The robot 1 may also be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. Since the robot 1 is a robot that exists in a real environment, it may also be called a real robot to distinguish it from a three-dimensional model virtual robot that exists in a virtual environment. The real environment refers to the environment in real space.

[0011] The robot control device 2 controls the operation of the robot 1 by a teaching playback method. The robot control device 2 that operates the robot 1 based on teaching data is already known, and detailed description thereof will be omitted. Note that the robot control system 100 may further include, for example, a teaching pendant for teaching the robot 1, and when the robot 1 is a welding robot, may further include a welding power source, a wire feeder, and the like, as necessary.

[0012] The teaching data correcting device 3 corrects the teaching data of the robot 1 while the robot 1 is in operation, by using a three-dimensional model of a virtual robot corresponding to the robot 1. The teaching data correcting device 3 will be described in detail later.

[0013] The display device 4 displays an image by superimposing it on an image of the real environment or the real environment itself. That is, the worker who corrects the teaching data can see both the real environment and the virtual environment images by the display device 4. The display device 4 may be a wearable display device that the worker who corrects the teaching data wears on his / her head, or may be a display device that is a portable information processing terminal such as a tablet terminal. The wearable display device may be, for example, a head-mounted display. The display device 4 may also have, for example, a transmissive display. In this case, the display device 4 displays an image by superimposing it on the real environment itself. As a wearable display device 4 having a transmissive display, for example, HoloLens (registered trademark) is known. The display device 4 having such a transmissive display can also be considered as a display device for realizing mixed reality (MR). The display device 4 may also have, for example, a non-transmissive display. In this case, the display device 4 displays an image by superimposing it on an image of the real environment. Therefore, it is preferable that the display device 4 having a non-transmissive display has a camera for photographing the real environment or is connected to a camera for photographing the real environment. An image of the real environment captured by the camera is displayed on a non-transmissive display in real time. For example, Oculus Quest is known as a wearable display device 4 having a non-transmissive display. Such a display device 4 having a non-transmissive display can be considered as a display device for realizing augmented reality (AR). The display device 4, which is a portable information processing terminal such as a tablet terminal, may have, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. In this embodiment, a case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

[0014] Fig. 2 is a block diagram showing the configuration of the teaching data correction device 3 according to this embodiment. As shown in Fig. 2, the teaching data correction device 3 is connected to the robot control device 2 and the display device 4, and includes a model storage unit 31, a teaching data storage unit 32, a positional relationship acquisition unit 33, a reception unit 34, an image generation unit 35, an output unit 36, a period reception unit 37, an area reception unit 38, and a correction unit 39.

[0015] The model storage unit 31 stores a three-dimensional model of a virtual robot corresponding to the robot 1, which is a real robot existing in a real environment. The three-dimensional model of the virtual robot is assumed to have the same size and configuration as the robot 1, except that it is configured as a three-dimensional model. Furthermore, this three-dimensional model is assumed to be able to change the angles of the joints of multiple arms, just like the robot 1.

[0016] There is no restriction on the process by which information on the three-dimensional model of the virtual robot is stored in the model storage unit 31. For example, information may be stored in the model storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the model storage unit 31. The model storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0017] The teaching data storage unit 32 stores teaching data of the robot 1, which is an actual robot. In this embodiment, the teaching data stored in the teaching data storage unit 32 is the same as the teaching data stored in the robot control device 2. In other words, the robot control device 2 and the teaching data correction device 3 each store the same teaching data. When the teaching data of one of the devices is corrected, the correction content is reflected in the other teaching data after the correction. The teaching data may be stored in only one of the robot control device 2 and the teaching data correction device 3. In this case, a device that does not store the teaching data may be able to access the teaching data of a device that stores the teaching data. For example, the teaching data storage unit 32 may be a storage unit in which the teaching data read by accessing the robot control device 2 is temporarily stored. The teaching data may indicate, for example, the position and posture of the robot 1 for each teaching point. The information indicating the position and posture of the robot 1 may be, for example, information indicating the position and posture of the hand of the robot 1, or information indicating the angle of each axis of the robot 1. The teaching data storage unit 32 is preferably realized by a non-volatile recording medium, but may be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

[0018] The positional relationship acquisition unit 33 acquires the relative positional relationship between the real environment in which the robot 1, which is a real robot, exists and the display device 4. Acquiring the relative positional relationship between the real environment and the display device 4 may, for example, be acquiring the relative positional relationship between a world coordinate system, which is a coordinate system of the real environment, and a display coordinate system, which is a local coordinate system of the display device 4. The relative positional relationship may be indicated, for example, by a homogeneous transformation matrix indicating the transformation between both coordinate systems. The relative positional relationship between the real environment and the display device 4 may be, for example, the relative positional relationship between the robot 1 or other objects existing in the real environment and the display device 4, or may be information indicating the position and attitude of the display device 4 in the real environment.

[0019] The method by which the positional relationship acquisition unit 33 acquires this relative positional relationship is not limited. For example, when an environmental map of the real environment is prepared, the positional relationship acquisition unit 33 may acquire the relative positional relationship, which is information indicating the position and orientation of the display device 4 in a three-dimensional real environment, using the environmental map of the real environment and the distance to the surrounding objects and the surrounding image acquired by the display device 4, using a method such as SLAM (Simultaneous Localization and Mapping) or Visual-SLAM. In this case, for example, the environmental map is held in the teaching data correction device 3, and the positional relationship acquisition unit 33 may acquire the relative positional relationship using the distance to the surrounding objects and the surrounding image received from the display device 4 and the environmental map. Also, for example, the acquisition of the relative positional relationship using the method such as SLAM or Visual-SLAM may be performed in the display device 4, and the relative positional relationship may be passed to the positional relationship acquisition unit 33. In this case, the acquisition of the relative positional relationship by the positional relationship acquisition unit 33 may be acceptance of the relative positional relationship. In this case, the display device 4 may have a depth sensor capable of measuring the distance to surrounding objects and a camera capable of acquiring images of the surroundings. The positional relationship acquisition unit 33 may receive an image captured by the camera of the display device 4, and acquire a homogeneous transformation matrix indicating the transformation between the world coordinate system and the display coordinate system using three or more feature points of the robot 1 or other objects included in the image.

[0020] The positional relationship acquisition unit 33 may acquire the relative positional relationship between the real environment and the display device 4 by a method other than the above. For an example of the method, refer to Patent Documents 1 to 3 described later. The relative positional relationship may be acquired by the positional relationship acquisition unit 33 as described above for each generation of a display image described later, or may not be acquired. In the latter case, for example, the relative positional relationship between the real environment and the display device 4 may be updated using a change in the position or attitude of the display device 4 after the relative positional relationship is acquired. This update may be performed by the positional relationship acquisition unit 33, for example. In addition, when this update is performed, the display device 4 may include, for example, an acceleration sensor or a sensor for acquiring the orientation of the display device 4, and the change in position or attitude may be acquired using these sensors. Note that the values ​​acquired by these sensors may be passed to a component that updates the relative positional relationship, such as the positional relationship acquisition unit 33. The sensor for acquiring the orientation of the display device 4 may be, for example, a gyro sensor or a direction sensor.

[0021] The reception unit 34 receives an operation on the three-dimensional model of the virtual robot. The reception unit 34 also receives a teaching instruction for the three-dimensional model of the virtual robot. The operation on the three-dimensional model may be, for example, an operation for changing the position or posture of at least a part of the three-dimensional model. The reception of the operation or teaching instruction may be, for example, performed via an input device such as a teaching pendant present in the real environment, or may be performed via a virtual input interface such as a virtual button or a virtual teaching pendant displayed on the display of the display device 4. The virtual input interface may be displayed on the display of the display device 4 in response to an operation such as an air tap, and when a button or the like is selected by the worker's finger or a pointing device, an input in response to the operation of the button or the like may be passed from the display device 4 to the reception unit 34. In addition, the position or posture of at least a part of the three-dimensional model of the virtual robot (for example, a hand tip of a tool or the like) may be changed by a gesture of the worker's hand or the like. In this case, for example, when the worker pinches the three-dimensional model displayed on the display with his / her hand (holding action), the part pinched with the hand is specified as the target of operation, and an operation to change the position and posture of the specified target is performed by changing the position and posture of the hand, and the operation on the specified target may be ended by ending the pinching action. In this case, for example, information indicating the target of operation (for example, information indicating the position or part in the three-dimensional model) and information indicating the content of the operation (for example, information indicating a change in position or a change in posture) may be passed from the display device 4 to the reception unit 34, or the result of hand tracking acquired by the display device 4 may be passed to the reception unit 34, and the target of operation and the content of the operation may be specified in the teaching data correction device 3. When the display device 4 acquires information indicating the target of operation and information indicating the content of the operation, the display device 4 may be able to access information on the current three-dimensional model of the virtual robot in the virtual space, which is held by the teaching data correction device 3.When an operation is performed using the worker's hand, the display device 4 may have a camera, and the position of the worker's hand on the display may be identified by performing hand tracking of the worker's hand photographed by the camera. Also, a method for converting an operation on a display image of a three-dimensional model according to a gesture of the worker's hand or the like into a change in the position or posture of the three-dimensional model in a three-dimensional virtual space is already known, and a detailed description thereof will be omitted.

[0022] The reception unit 34 may receive information input from, for example, an input device or the display device 4, or may receive information transmitted via a wired or wireless communication line. The reception unit 34 may or may not include a device for receiving the information (for example, an input device or a communication device). The reception unit 34 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0023] The image generating unit 35 generates a display image for displaying a three-dimensional model so that the robot 1, which is a real robot, and the three-dimensional model of the virtual robot have a predetermined positional relationship based on the three-dimensional model stored in the model storage unit 31 and the relative positional relationship between the real environment and the display device 4 acquired by the positional relationship acquiring unit 33. When an operation on the virtual robot is accepted, the image generating unit 35 generates a display image for displaying the operated three-dimensional model. The predetermined positional relationship may be, for example, a positional relationship determined in advance, or may be a positional relationship that can be changed by the operator. The three-dimensional model of the virtual robot may be displayed, for example, at the position of the robot 1, or at a position different from the robot 1. Displaying the three-dimensional model of the virtual robot at the position of the robot 1 means displaying the three-dimensional model so that the virtual robot is placed at the same position as the robot 1, and may be, for example, displaying the end portion on the base end side of the three-dimensional model of the virtual robot (for example, a part attached to a floor surface, etc.) so as to overlap the end portion on the base end side of the robot 1. In addition, when the three-dimensional model of the virtual robot is displayed at the position of the robot 1 when the teaching data is corrected, the three-dimensional model may be operated, for example, using a teaching pendant in the real environment or a virtual teaching pendant. This is because when the three-dimensional model is operated by the operator's hand gestures, there is a possibility that the operator may approach the position of the virtual robot, i.e., the position of the robot 1, which is dangerous. When the three-dimensional model of the virtual robot is displayed at a position different from the robot 1, for example, the three-dimensional model may be displayed next to the robot 1. In either case, the three-dimensional model of the virtual robot is displayed so that the positional relationship with the robot 1, which is the real robot, does not change. Therefore, even if the operator changes the orientation of the display device 4, the display position of the three-dimensional model of the virtual robot in the real environment does not change.

[0024] The position of the robot 1 in the world coordinate system is known. In addition, the positional relationship between the robot 1 and the three-dimensional model of the virtual robot is also determined. Therefore, by using these pieces of information, the image generation unit 35 can place the three-dimensional model of the virtual robot so as to have a predetermined positional relationship with the position of the robot 1 in the virtual space in which the position of the robot 1 is determined in advance. The angles of each joint of the virtual robot are set to initial values ​​when no operation is performed, and are set to values ​​after the operation when an operation is performed. The angles of each joint after the operation may be calculated by inverse kinematics using the position and orientation of the hand in the three-dimensional model of the virtual robot after the operation, for example, as with a real robot. In addition, the image generation unit 35 can know the relationship between the world coordinate system and the display coordinate system, which is the local coordinate system of the display device 4, based on the relative positional relationship acquired by the positional relationship acquisition unit 33, and therefore can specify the position and orientation of the display device 4 in the virtual space. Therefore, the image generation unit 35 can generate a two-dimensional display image for displaying the three-dimensional model by rendering the three-dimensional model of the virtual robot in the virtual space based on the position and orientation of the display device 4. When the virtual robot is operated, the shape of the three-dimensional model of the virtual robot in the virtual space is changed accordingly, as described above. In addition, when the position or orientation of the display device 4 is changed in the real environment, the position and direction of the viewpoint in the virtual space are changed accordingly. Then, rendering is performed after the change, and a display image of the three-dimensional model after the operation and a display image of the three-dimensional model after the change in the position or orientation of the display device 4 are generated. Note that the image generating unit 35 generates a display image such that the size of the display image of the three-dimensional model of the virtual robot is consistent with the real environment when the display image is displayed on the display of the display device 4. That is, the display image is generated so that the three-dimensional model of the virtual robot displayed on the display of the display device 4 and the real robot arranged in the real environment so as to have the same relative positional relationship as the three-dimensional model are the same size when viewed through the display device 4.

[0025] For example, when the teaching data is corrected, the image generating unit 35 may generate a display image for displaying a three-dimensional model of the virtual robot at a position different from that of the actual robot, robot 1. For example, if the virtual robot is displayed at the same position as the robot 1 while the robot 1 is in operation, it may be difficult for the worker to see the virtual robot. This is particularly true when the robot 1 is a welding robot and welding is being performed. For this reason, it is preferable to display the virtual robot at a position different from that of the robot 1 when the teaching data is corrected.

[0026] For example, after correcting the teaching data, the image generating unit 35 may generate a display image for displaying a three-dimensional model of a virtual robot that operates according to the corrected teaching data. In this case, the image generating unit 35 may generate a plurality of display images of the virtual robot at a predetermined time interval based on the corrected teaching data. For example, the display image may be for displaying the three-dimensional model of the virtual robot at the position of the robot 1, which is a real robot, or may be for displaying the three-dimensional model of the virtual robot at a position different from the robot 1. In the former case, the movement according to the corrected teaching data can be confirmed at the position where the robot 1 is placed in the real environment.

[0027] Although the display image generated by the image generating unit 35 is usually a still image, when a plurality of display images are generated in succession within a short period of time, a display image substantially similar to a moving image is generated. For example, a display image for displaying a three-dimensional model of a virtual robot that operates according to corrected teaching data is usually a display image similar to a moving image.

[0028] In addition, it may not be possible to operate the virtual robot according to the accepted operation. For example, an operation to move the hand of the virtual robot beyond the movable range or rotate the hand beyond the rotatable range may be accepted. In such a case, the image generating unit 35 may not generate a display image according to the operation, or may generate a display image of a three-dimensional model of the virtual robot that has been moved or rotated within the possible range.

[0029] The output unit 36 ​​outputs the display image generated by the image generation unit 35 to the display device 4. The output unit 36 ​​may output only the display image. In this case, the display image is displayed on the display device 4 by being superimposed on the image of the real environment or the real environment itself. On the other hand, when the display device 4 has a non-transparent display and the image of the real environment captured by the display device 4 is accepted by the teaching data correction device 3, the result of combining the image of the real environment and the display image may be output to the display device 4. This combination may be performed, for example, by a combination unit (not shown) included in the teaching data correction device 3.

[0030] When the 3D model of the virtual robot is displayed at a position different from that of the actual robot, robot 1, for example, as shown in Fig. 1, the 3D model 10 of the virtual robot may be displayed next to the robot 1. The robot 1 and the 3D model 10 of the virtual robot in Fig. 1 are shown typically as viewed by an operator performing teaching via a display device 4.

[0031] Note that displaying a three-dimensional model of a virtual robot together with a real robot and manipulating the three-dimensional model of the virtual robot using a virtual input interface of a display device or a worker's gestures, etc., are already publicly known, as shown in, for example, the following Patent Documents 1 to 3, and detailed explanations thereof will be omitted. [Prior art documents] [Patent documents]

[0032] [Patent Document 1] JP 2017-100234 A [Patent Document 2] JP 2020-055075 A [Patent Document 3] JP 2020-069538 A

[0033] The period receiving unit 37 receives a period for which the teaching data is to be corrected during the operation of the robot 1, which is a real robot. The period received by the period receiving unit 37 may specify a period in the operation of the robot 1. This reception may be, for example, reception of the end of the period and the time length of the period. In this case, for example, when an operator who corrects the teaching data watches the operation of the robot 1 and finds a part of the teaching data that he / she wants to correct, he / she may input information indicating that the current time is the end of the period at the end of the operation of the part to be corrected, and may input the time length of the period to be corrected. In addition, this reception may be, for example, reception of the start and end of the period. In this case, for example, when an operator who corrects the teaching data watches the operation of the robot 1 and finds a part of the teaching data that he / she wants to correct, he / she may input information indicating that the current time is the start of the period and information indicating that the current time is the end of the period at the start and end of the operation of the part to be corrected, respectively. The information input by the operator in this manner is accepted by the period receiving unit 37.

[0034] The area receiving unit 38 receives an area to be corrected for the teaching data during the operation of the robot 1, which is an actual robot. The area received by the area receiving unit 38 may specify an area in the operation of the robot 1. This reception may be, for example, reception of a designation of an area of ​​the position of the hand of the robot 1. In this case, for example, when an operator who corrects the teaching data watches the operation of the robot 1 and finds a part of the teaching data that he / she wants to correct, he / she may input the position of the hand at the time of the operation of the part to be corrected. This input may be performed, for example, by specifying an area in space using the display device 4 or the like. For example, when the robot 1 is a welding robot, information indicating a welded part by the operator's finger or pointing device may be input by the area receiving unit 38. The hand of the robot 1 may be, for example, a TCP (tool center point).

[0035] The period acceptance unit 37 and the area acceptance unit 38 may accept information input from, for example, an input device or the display device 4, or may receive information transmitted via a wired or wireless communication line. Note that the period acceptance unit 37 and the area acceptance unit 38 may or may not include a device for accepting (for example, an input device or a communication device). Furthermore, the period acceptance unit 37 and the area acceptance unit 38 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0036] The correction unit 39 corrects the teaching data according to the three-dimensional model of the virtual robot when the reception unit 34 receives a teaching instruction for the three-dimensional model of the virtual robot during the operation of the robot 1, which is a real robot. That is, information indicating the position and posture of the three-dimensional model displayed when the teaching instruction is received is accumulated in the teaching data storage unit 32 as the corrected teaching data. For example, when a period to be corrected is received by the period reception unit 37, the correction unit 39 may correct the teaching data for the period to be corrected. The teaching data for the period to be corrected may be the teaching data used by the robot 1, which is a real robot, for the operation of the period. Furthermore, for example, when a region to be corrected is received by the region reception unit 38, the correction unit 39 may correct the teaching data for the region to be corrected. The teaching data for the region to be corrected may be the teaching data used by the robot 1, which is a real robot, for the operation of the region. Specifically, the correction unit 39 may specify teaching data corresponding to the period or area to be corrected, and may sequentially correct the teaching points specified by the specified teaching data in response to the accepted operation. When the period to be corrected is accepted, the correction unit 39 may specify teaching data corresponding to the operation of the period, for example, by accessing the robot control device 2. Also, when the area to be corrected is accepted, the correction unit 39 may specify teaching data in which the area is the position of the hand of the robot 1. For example, the correction unit 39 may cause the image generation unit 35 to generate a display image of the three-dimensional model of the virtual robot in a position and posture corresponding to the teaching points included in the specified teaching data to be corrected. Then, the position and posture of the three-dimensional model of the virtual robot may be changed by accepting an operation on the three-dimensional model of the virtual robot. Thereafter, in response to accepting an instruction for instruction, the correction unit 39 may correct the information of the teaching points to be corrected to new information corresponding to the position and posture of the three-dimensional model of the virtual robot at that time. In this manner, the correction of the teaching data may be performed, for example, for each teaching point.Alternatively, the correction unit 39 may obtain new teaching data to replace the specified teaching data to be corrected by receiving multiple times the operation of the three-dimensional model of the virtual robot and the instruction for teaching, and change the teaching data to be corrected to the new teaching data. The correction unit 39 may obtain information required for correcting the teaching data, such as the position and posture of the hand in the robot coordinate system, which is the local coordinate system of the three-dimensional model of the virtual robot, or information such as the angle of each joint, from the image generation unit 35, or from another configuration. The position and posture of the hand in the robot coordinate system of the three-dimensional model of the virtual robot can be obtained, for example, by converting the position and posture of the hand in the virtual space using a homogeneous transformation matrix indicating the transformation between the virtual space and the robot coordinate system of the virtual robot.

[0037] The model storage unit 31 and the teaching data storage unit 32 may be realized by the same recording medium or by separate recording media. In the former case, for example, an area storing a 3D model of a virtual robot becomes the model storage unit 31, and an area storing teaching data becomes the teaching data storage unit 32.

[0038] Furthermore, the teaching data corrected by the correcting unit 39 may be accessible from, for example, the robot control device 2, or the teaching data correcting device 3 may include an output unit (not shown) for outputting the corrected teaching data to the robot control device 2. The output may be, for example, a transmission.

[0039] Next, the operation of the teaching data correcting device 3 will be described with reference to the flowchart of FIG.

[0040] (Step S101) The period receiving unit 37 judges whether or not the period to be corrected of the teaching data has been received. If the period to be corrected of the teaching data has been received, the process proceeds to step S102, and if not, the process proceeds to step S103.

[0041] (Step S102) The correction unit 39 identifies teaching data to be corrected according to the accepted period to be corrected.

[0042] (Step S103) The area receiving unit 38 judges whether or not the area of ​​the teaching data to be corrected has been received. If the area of ​​the teaching data to be corrected has been received, the process proceeds to step S104. If not, the process returns to step S101.

[0043] (Step S104) The correction unit 39 identifies teaching data to be corrected according to the accepted area to be corrected.

[0044] (Step S105) The correction unit 39 corrects the identified teaching data. This process will be described later with reference to the flowchart of FIG.

[0045] (Step S106) The positional relationship acquisition unit 33 acquires the relative positional relationship between the real environment and the display device 4.

[0046] (Step S107) The image generating unit 35 uses the 3D model of the virtual robot, the relative positional relationship acquired in step S106, and the corrected teaching data to generate a display image of the 3D model according to the operation of the virtual robot according to the corrected teaching data. The image generating unit 35 may generate a plurality of display images at predetermined time intervals, for example. Note that the image generating unit 35 may generate a display image for displaying the 3D model of the virtual robot at the position of the robot 1, for example.

[0047] (Step S108) The output unit 36 ​​outputs the generated display image to the display device 4. The output unit 36 ​​may sequentially output a plurality of display images to the display device 4 at predetermined time intervals. In this manner, the operator can check the operation of the virtual robot according to the corrected teaching data. Then, the process returns to step S101.

[0048] Note that the order of the processes in the flowchart of Fig. 3 is an example, and the order of the steps may be changed as long as the same results can be obtained. In addition, in the flowchart of Fig. 3, the process ends when the power is turned off or an interrupt is issued to end the process.

[0049] FIG. 4 is a flowchart showing details of the process of correcting the teaching data (step S105) in the flowchart of FIG.

[0050] (Step S201) The modification unit 39 sets a counter i to 1.

[0051] (Step S202) The positional relationship acquisition unit 33 acquires the relative positional relationship between the real environment and the display device 4.

[0052] (Step S203) The image generation unit 35 uses the 3D model of the virtual robot and the relative positional relationship acquired in step S202 to generate a display image corresponding to the position and posture of the 3D model of the virtual robot according to the i-th teaching point in the teaching data to be corrected.

[0053] (Step S204) The output unit 36 ​​outputs the display image generated in step S203 to the display device 4. As a result, the operator correcting the teaching data can see a three-dimensional model of the virtual robot in a position and posture according to the i-th teaching point.

[0054] (Step S205) The reception unit 34 judges whether or not an operation for the three-dimensional model of the virtual robot has been received. If an operation for the three-dimensional model of the virtual robot has been received, the process proceeds to step S206, and if not, the process proceeds to step S209.

[0055] (Step S206) The positional relationship acquisition unit 33 acquires the relative positional relationship between the real environment and the display device 4.

[0056] (Step S207) The image generation unit 35 generates a display image for displaying a three-dimensional model of the virtual robot whose position and posture have been changed by the accepted operation, using the relative positional relationship acquired by the positional relationship acquisition unit 33.

[0057] (Step S208) The output unit 36 ​​outputs the generated display image to the display device 4. Then, the process returns to step S205. In this way, the worker who corrects the teaching data can check the virtual robot whose position and posture have been changed. If the worker is allowed to perform new teaching based on the position and posture of the virtual robot, he / she inputs a teaching instruction, and if not, he / she inputs the operation of the virtual robot again. Note that when the position or orientation of the display device 4 is changed, it is preferable that a new display image is generated accordingly. Therefore, even if no operation is being performed, the processes of steps S206 to S208 may be performed repeatedly.

[0058] (Step S209) The reception unit 34 determines whether or not a teaching instruction has been received. If a teaching instruction has been received, the process proceeds to step S210, and if not, the process returns to step S205.

[0059] (Step S210) The correction unit 39 accumulates teaching data corresponding to the position and posture of the three-dimensional model of the virtual robot at that time as teaching data corresponding to the i-th teaching point in the teaching data storage unit 32. This accumulation may be, for example, accumulation by overwriting, or it may not be the same. In the latter case, it is preferable that the latest teaching data can be distinguished from other teaching data.

[0060] (Step S211) The modification unit 39 increments the counter i by one.

[0061] (Step S212) The correction unit 39 judges whether the teaching data to be corrected includes the i-th teaching point. If the i-th teaching point is included, the process returns to step S202. If not, the series of corrections of the teaching data is completed, and the process returns to the flowchart of FIG.

[0062] Next, the operation of the robot control system 100 according to this embodiment will be described using a specific example.

[0063] Assume that an operator wears a display device 4 having a transmissive display on his head and watches the motion of the robot 1, which is a real robot. At this time, a three-dimensional model of the virtual robot may or may not be displayed. Then, when there is a part of the motion of the robot 1 that the operator wishes to correct, the operator operates an input device (not shown) at the time when the motion of the part to be corrected is completed, and inputs indicating the end of the period of the teaching data to be corrected. After that, the operator inputs the length of the period (for example, 15 seconds or 30 seconds). The input indicating the timing of the end of the period and the input indicating the length of the period are received by the period receiving unit 37, and are respectively passed to the correction unit 39 (step S101).

[0064] When the correction unit 39 receives an input indicating the end of the period to be corrected for the teaching data, it acquires the time T2 at that time from a clock unit (not shown). When the correction unit 39 receives the time length of the period, it specifies the start time T1 of the period to be corrected, which is a time point earlier than the time T2 by the time length. Then, the correction unit 39 accesses the robot control device 2 and specifies the teaching data used in the operation of the robot 1 from the time T1 to the time T2 (step S102). For example, if the tip side 1a of the robot 1 moves as shown in FIG. 5A from the time T1 to the time T2, the teaching data corresponding to the movement is specified. Note that this time may be, for example, information indicating a time point in the operation of the robot 1 (for example, a timer value, etc.).

[0065] Next, the correction unit 39 performs a process for correcting the specified teaching data (step S105). Specifically, a three-dimensional model of the virtual robot is displayed so that the position and posture correspond to the first teaching point included in the specified teaching data to be corrected (steps S201 to S204). Then, if the operator wants to change the teaching point, the operator changes the position and posture of the three-dimensional model of the virtual robot as shown in FIG. 6. This change may be performed, for example, by the operator holding the tip side 1a of the three-dimensional model of the virtual robot shown by the solid line in FIG. 6 with his index finger and thumb, moving it to the position shown by the dashed line, and releasing the hold. This operation is accepted by the acceptance unit 34, and the three-dimensional model after the operation is displayed (steps S205 to S208). Then, if it is OK to perform teaching with the changed position and posture, the operator selects the virtual teaching button displayed by air tapping with his finger. In response to the selection, a teaching instruction is received by the reception unit 34, and teaching data corresponding to the position and posture of the three-dimensional model of the virtual robot at that time is stored in the teaching data storage unit 32 (steps S209, S210). In this manner, the first teaching point is rewritten.

[0066] On the other hand, if there is no need to change the first teaching point, the operator selects the virtual teaching button when the display corresponding to the first teaching point is displayed. Then, teaching data having the same contents as the first teaching point already stored is stored in the teaching data storage unit 32 (steps S209, S210). Note that, if a teaching instruction is accepted without performing an operation on the 3D model, instead of storing the same teaching data, the process of storing the teaching data corresponding to the first teaching point may be skipped.

[0067] Thereafter, the three-dimensional model of the virtual robot is displayed so that the position and posture correspond to the second teaching point included in the teaching data to be corrected (steps S211, S212, S202 to S204). In this manner, the second and subsequent teaching points are also changed sequentially as necessary. Then, when all teaching points included in the teaching data to be corrected have been appropriately changed as necessary, the process of correcting the teaching data ends. A display image showing the operation of the virtual robot according to the corrected teaching data is displayed on the display of the display device 4 (steps S106 to S108). This display allows the operator to check the corrected teaching data.

[0068] Here, the case where the range of the teaching data to be corrected is specified by a period has been described, but the range of the teaching data to be corrected may be specified by a region. In this case, for example, when the robot 1, which is a real robot, is operating, the region R1 of the teaching data to be corrected may be received by the region receiving unit 38 as shown in FIG. 5B (step S103). In this case, for example, the region R1 including the trajectory of the tip of the tool corresponding to the teaching data to be corrected may be specified by the user's finger or pointing device, and the region R1 may be received by the region receiving unit 38. Since the region R1 is usually information indicating a region in the display coordinate system or the world coordinate system, it may be converted into information indicating a region in the local coordinate system of the robot 1. This conversion may be performed by the correction unit 39 or the like using, for example, the relative positional relationship acquired by the positional relationship acquisition unit 33 and the placement position of the robot 1 in the real environment (for example, a homogeneous transformation matrix indicating the transformation between the world coordinate system and the local coordinate system of the robot 1). Then, the correction unit 39 may use the teaching data to specify the range of the teaching data in which the tip of the tool is the region (step S104), and correct the teaching data for the specified range (step S105). The processing after the teaching data to be corrected is specified is the same as described above, and detailed description thereof will be omitted.

[0069] As described above, according to the teaching data correction device 3 of this embodiment, even if the robot 1, which is a real robot, is in operation, the teaching data of the robot 1 can be corrected using a virtual robot without stopping the operation of the robot 1 midway. For example, in multi-layer welding, the playback of the teaching data continues for a long time, so if the correction of the teaching data is performed after waiting for the end of the playback, productivity will decrease. On the other hand, in the teaching data correction device 3 of this embodiment, the teaching data can be corrected while the robot 1 is in operation, so that the robot 1 can immediately start the playback operation using the corrected teaching data after the playback operation ends, thereby improving productivity. In addition, by accepting the period and area to be corrected of the teaching data, the range of the teaching data to be corrected can be easily specified. This is because it is only necessary to specify the period of the operation to be corrected and the area of ​​the operation to be corrected in the operation of the robot 1. In addition, when correcting the teaching data, the three-dimensional model of the virtual robot is displayed at a position different from that of the robot 1, so that the three-dimensional model of the virtual robot does not overlap with the operating robot 1, and the correction of the teaching data can be performed more easily. In addition, after the teaching data is corrected, a three-dimensional model of the virtual robot operating according to the corrected teaching data is displayed at the position of robot 1, making it possible to confirm the operation according to the corrected teaching data at the position of robot 1.

[0070] In the present embodiment, the case where the teaching data correction device 3 has the period acceptance unit 37 and the area acceptance unit 38 has been described, but this is not necessary. The teaching data correction device 3 does not have to have at least one of the period acceptance unit 37 and the area acceptance unit 38. When the teaching data correction device 3 does not have both the period acceptance unit 37 and the area acceptance unit 38, the teaching data to be corrected may be manually specified by the user. For example, the teaching data itself may be displayed on the display device 4, and a portion of it to be corrected may be specified by the user. In this case, while the robot 1 is performing the task A, the teaching data correction device 3 may correct the teaching data related to the task B.

[0071] In the above embodiment, each process or function may be realized by centralized processing by a single device or a single system, or may be realized by distributed processing by multiple devices or multiple systems. For example, at least a part of the configuration of the teaching data correction device 3 may be physically included in a device having a display, a robot controller, etc. Therefore, the division of the devices shown in FIG. 2 may be considered to be for convenience according to function, not according to physical device.

[0072] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0073] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0074] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0075] In addition, in the above embodiment, when two or more components included in the teaching data correction device 3 have a communication device, an input device, etc., the two or more components may have a physically single device or may have separate devices.

[0076] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. During execution, the program execution unit may execute the program while accessing a storage unit or a recording medium. The program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). The program may be used as a program constituting a program product. The computer that executes the program may be single or multiple. That is, centralized processing may be performed, or distributed processing may be performed.

[0077] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Explanation of symbols]

[0078] REFERENCE SIGNS LIST 1 robot, 2 robot control device, 3 teaching data correction device, 4 display device, 31 model storage unit, 32 teaching data storage unit, 33 positional relationship acquisition unit, 34 reception unit, 35 image generation unit, 36 output unit, 37 period reception unit, 38 area reception unit, 39 correction unit, 100 robot control system

Claims

1. a model storage unit for storing a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment; a teaching data storage unit in which teaching data of the actual robot is stored; a reception unit that receives an instruction for operation and instruction for the three-dimensional model of the virtual robot; a positional relationship acquisition unit that acquires a relative positional relationship between a real environment in which the real robot exists and a display device that displays an image of the real environment or the real environment itself by superimposing the image; an image generating unit that generates a display image for displaying a manipulated three-dimensional model so as to have a predetermined positional relationship with the actual robot, based on the three-dimensional model and the relative positional relationship; an output unit that outputs the display image to the display device; a period receiving unit that receives a period of the teaching data to be corrected during the operation of the real robot, the period being a period in the operation of the real robot; a correction unit that corrects the teaching data used by the real robot for operation during the period to be corrected in accordance with the three-dimensional model of the virtual robot when a teaching instruction for the three-dimensional model of the virtual robot is accepted by the accepting unit during operation of the real robot.

2. A model storage unit in which a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment is stored; a teaching data storage unit in which teaching data of the actual robot is stored; a reception unit that receives an instruction for operation and instruction for the three-dimensional model of the virtual robot; a positional relationship acquisition unit that acquires a relative positional relationship between a real environment in which the real robot exists and a display device that displays an image of the real environment or the real environment itself by superimposing the image; an image generating unit that generates a display image for displaying a manipulated three-dimensional model so as to have a predetermined positional relationship with the actual robot, based on the three-dimensional model and the relative positional relationship; an output unit that outputs the display image to the display device; an area receiving unit that receives an area to be corrected of teaching data, the area being an area in the motion of the real robot, during the motion of the real robot; a correction unit that corrects the teaching data used by the real robot for operation in the area to be corrected in accordance with the three-dimensional model of the virtual robot when a teaching instruction for the three-dimensional model of the virtual robot is accepted by the accepting unit while the real robot is in operation.

3. A model storage unit in which a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment is stored; a teaching data storage unit in which teaching data of the actual robot is stored; a reception unit that receives an instruction for operation and instruction for the three-dimensional model of the virtual robot; a positional relationship acquisition unit that acquires a relative positional relationship between a real environment in which the real robot exists and a display device that displays an image of the real environment or the real environment itself by superimposing the image; an image generating unit that generates a display image for displaying a manipulated three-dimensional model so as to have a predetermined positional relationship with the actual robot, based on the three-dimensional model and the relative positional relationship; an output unit that outputs the display image to the display device; a correction unit that corrects the teaching data in accordance with the three-dimensional model of the virtual robot when a teaching instruction for the three-dimensional model of the virtual robot is accepted by the acceptance unit during operation of the real robot, The image generation unit generates a display image for displaying a three-dimensional model of the virtual robot at a position different from that of the actual robot when correcting the teaching data.

4. 4. The teaching data correction device according to claim 3, wherein the image generation unit generates a display image for displaying a three-dimensional model of the virtual robot operating in accordance with the corrected teaching data at the position of the actual robot after the teaching data is corrected.

Images