Control devices and robot systems

Abstract

A control device includes: a correction calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from a vision sensor mounted on a mobile robot; and a control unit that changes at least one of the camera position and camera posture of the aforementioned vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor.

Description

Technical Field

[0001] This invention relates to a robot control technology, and more particularly to a control device and robot system for applying corrections to the robot's movements. Prior Technology

[0002] Background Technology

[0003] In recent years, many robot systems have been proposed for mobile robots, such as robots mounted on trolleys, AGVs (automated guided vehicles), transport rail systems, or robots integrated with transport devices. These robot systems enable mobile robots to move to the vicinity of machine tools, construction machinery, or workpieces such as vehicles, aircraft, buildings, and their parts, and to perform various tasks using the robots.

[0004] When a mobile robot performs various tasks such as loading / unloading workpieces, changing tools, and processing workpieces (e.g., cutting, grinding, welding, fitting, fastening, sealing, etc.), its position or posture relative to the workspace changes when the robot's stopping position or posture changes, or when the position or posture of the machinery or workpiece acting as the work object changes. Therefore, the robot cannot perform its tasks appropriately simply by repeating the same actions each time. Consequently, a technique has been proposed that measures the deviation of the robot's position or posture relative to the workspace and applies corrections to the robot's movements.

[0005] Regarding methods for applying corrections to robot movements, one technique involves installing a vision sensor at the robot's end cap, using the vision sensor to detect the three-dimensional positions of multiple reference points set in the workspace, calculating the deviation (correction amount) from the robot's reference position or posture relative to the workspace, and applying corrections to the robot's movements (e.g., Patent Documents 1 and 2). However, situations exist where, for example, the mobile robot moves near other machinery or workpieces, causing a change in its stopping position or posture, or when the position or posture of the machinery or workpiece in the workspace changes, the reference points in the workspace may be outside the field of view of the vision sensor. Therefore, to bring the reference points into the field of view of the vision sensor, manual changes to the mobile robot's stopping position or posture, manual changes to the position or posture of the machinery or workpiece, or re-teaching of the camera position or posture are required, which involve labor or trial and error. Consequently, the robot's correction processing takes time.

[0006] Patent document 1 describes a method for correcting the robot’s handling position on a workpiece by fixing a vision sensor on an unmanned transport vehicle equipped with a robot and setting a mark on the worktable. When the unmanned transport vehicle is stopped at the teaching position, the robot’s handling position on the workpiece is corrected based on the position deviation of the mark detected by the vision sensor.

[0007] Patent Document 2 describes that: if the position of the reference point is detected by a camera mechanism when the unmanned transport vehicle equipped with the robot is stopped, and the robot performs a predetermined operation on the workpiece in the stopped state, the operation time will be longer and it will be difficult to achieve high productivity. Therefore, when the first moving body equipped with the robot is moving, two or more images of the reference object are obtained by the camera installed on the first moving body at different times, and the predetermined operation is performed based on the two or more images while the first moving body is moving. Prior technology documents Patent documents

[0008] [Patent Document 1] Japanese Patent Application Publication No. 11-156764 [Patent Document 2] Japanese Patent Application Publication No. 2019-093481 Summary of the Invention

[0009] Invention Summary The problem the invention aims to solve

[0010] In view of the aforementioned problems, the object of the present invention is to provide a technique for automating the correction process of robots. The means to solve the problem

[0011] One aspect of this disclosure is to provide a control device comprising: a correction calculation unit that calculates a correction amount for the robot's movement relative to the workspace from information from a vision sensor mounted on a mobile robot; and a control unit that changes at least one of the camera position and camera posture of the aforementioned vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor. Another aspect disclosed herein provides a robot system comprising: a mobile robot; a vision sensor mounted on the robot; a correction calculation unit that calculates a correction amount for the robot's movement relative to the workspace from information from the vision sensor; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor. Invention Effects

[0012] According to the present disclosure, even when the stopping position or stopping posture of the mobile robot changes, or when the position or posture of the machine or workpiece being worked on changes, the camera position and camera posture of the vision sensor are automatically changed based on the correction amount. Therefore, the robot's correction processing can be automated without the labor or trial and error of manually changing the stopping position or stopping posture of the mobile robot, or manually changing the position or posture of the machine or workpiece being worked on, or re-teaching the camera position or camera posture in order to bring the reference point into the field of view of the vision sensor. Simple Explanation of the Diagram

[0013] Figure 1 is a structural diagram of the robot system in the first embodiment. Figure 2 is a functional block diagram of the robot system in the first embodiment. Figure 3 is a flowchart of the first correction process for the robot system in the first embodiment. Figure 4 is a flowchart of the next and subsequent corrections of the robot system in the first embodiment. Figure 5 is an illustrative diagram illustrating an example of how a vision sensor detects three-dimensional position by comparing the camera position and orientation with a reference point. Figure 6 is an explanatory diagram of an example of calculating the deviation (correction) of a mobile robot. Figure 7 is a flowchart of the correction process for the robot system in the second embodiment. Figure 8 is an explanatory diagram of the calculation principle of the robot's movement (correction amount) in the second embodiment. Implementation

[0014] Forms used to implement inventions

[0015] The embodiments disclosed herein will now be described in detail with reference to the accompanying drawings. In each drawing, the same or similar symbols are used for the same or similar constituent elements. Furthermore, the embodiments described below are not intended to limit the technical scope of the invention described in the claims or the meaning of the terms used.

[0016] The robot system 1 of the first embodiment will now be described. Figure 1 is a structural diagram of the robot system 1 of the first embodiment. The robot system 1 includes a mobile robot 10, a vision sensor 11, and a control device 12 for controlling the mobile robot 10. Furthermore, the robot system 1 also includes a teaching device 13 for teaching or confirming the status of the mobile robot 10, but this is not necessary.

[0017] The mobile robot 10 includes a robot 10a, a conveying device 10b, and a tool 10c. The mobile robot 10 is configured to detachably connect the robot 10a and the conveying device 10b, but it is not limited to this. In other embodiments, the robot 10a and the conveying device 10b can also be integrated.

[0018] Robot 10a is composed of a multi-joint robot, but is not limited to this. In other embodiments, it may also be composed of other industrial robots (robotic arms) such as single-joint robots, dual-arm robots, and parallel robots, or robots of other forms such as humanoid robots. Robot 10a is mounted on a conveying device 10b and is controlled by a control device 12.

[0019] The conveying device 10b is composed of a manual conveying device such as a trolley, but it is not limited to this. In other embodiments, it can also be composed of an automated conveying device such as an AGV (automated guided vehicle) or a conveying track system. When it is an automated conveying device, the conveying device 10b can also be controlled by the control device 12.

[0020] Tool 10c is composed of a multi-finger gripping hand, an adhesive hand, or other hand parts, but is not limited to these. In other embodiments, it can also be composed of a cutting tool, a welding tool, a sealing tool, or other processing tool. Tool 10c is detachably connected to the front end of the hand of robot 10a.

[0021] The vision sensor 11 has a two-dimensional sensor that outputs brightness information, but is not limited to this; in other embodiments, it may also have a three-dimensional sensor that outputs distance information. Furthermore, the vision sensor 11 is composed of one camera, but is not limited to this; in other embodiments, it may be composed of two stereo cameras. The vision sensor 11 is mounted on the mobile robot 10. The vision sensor 11 is mounted on the wrist of the robot 10a, but is not limited to this; in other embodiments, it may also be mounted on the arm of the robot 10a, the tool 10c, the conveying device 10b, or other movable parts.

[0022] The control device 12 is composed of a known PLC (programmable logic controller), but in other embodiments it can also be composed of other computers. The control device 12 includes a processor, memory, input / output interfaces, etc. (not shown), interconnected by buses. The control device 12 controls the movements of the mobile robot 10 according to the action program taught by the teaching device 13. The control device 12 controls the movements of the robot 10a, but is not limited to this; in other embodiments it can also control the movements of the conveying device 10b.

[0023] The control device 12 is configured with various coordinate systems, such as a world coordinate system, a mechanical coordinate system, a flange coordinate system, a tool coordinate system, a camera coordinate system, and a user coordinate system. These coordinate systems can be, for example, orthogonal coordinate systems. For ease of explanation, in this embodiment, the control device 12 is configured with both a mechanical coordinate system M and a camera coordinate system C. The mechanical coordinate system M is fixed at a reference position (e.g., a base) of the robot 10a. The camera coordinate system C is fixed at a reference position (e.g., a focal point) of the vision sensor 11.

[0024] The teaching device 13 is composed of a teaching aid, but is not limited to this; in other implementations, it may also be composed of a teaching control panel, other computers, etc. The teaching device 13 edits or creates the motion program of the mobile robot 10. The teaching device 13 sends the edited or created motion program to the control device 12.

[0025] In the robot system 1 described above, the mobile robot 10 moves to the vicinity of the machine 20, which is the object of the operation, to perform loading / unloading of the workpiece W, tool changing of the machine 20, etc., but is not limited to this. In other embodiments, it may also perform other operations such as processing of the workpiece W (e.g., cutting, grinding, welding, fitting, fastening, sealing, etc.).

[0026] Machine 20 is composed of known tooling mechanisms such as milling machines, but is not limited to these. In other embodiments, it may also be composed of other industrial machinery such as construction machinery and agricultural machinery. For example, machine 20 has a tool 21 for processing workpiece W and a control device 22 for controlling the movement of tool 21. The control device 22 is composed of a known CNC (computerized numerical control) device.

[0027] The mobile robot 10 is located in the workspace S and performs tasks such as loading / unloading workpieces W onto or onto machinery 20, or changing tools 21 on machinery 20. When the mobile robot 10 moves toward other machinery 20 or other workpieces W, causing the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) to change, or when the position or posture of the machinery 20 or workpiece W in the workspace S that is the object of the work changes, since at least one of the position and posture of the mobile robot 10 relative to the workspace S will also change, the robot 10a cannot properly perform the work simply by performing the same action each time.

[0028] Therefore, the control device 12 calculates the deviation of the mobile robot 10 from the information of the vision sensor 11 from at least one of its position and posture relative to the workspace S as a correction amount, and applies correction to the movement of the mobile robot 10 (robot 10a) based on the deviation amount (correction amount). Although the control device 12 applies correction to the movement of the robot 10a based on the deviation amount (correction amount), when the conveying device 10b is an automatic conveying device, there is also a situation where the movement of the conveying device 10b is corrected based on the deviation amount (correction amount).

[0029] To calculate the deviation of the mobile robot 10 from at least one of its position and posture relative to the workspace S, the control device 12 detects the three-dimensional positions of reference points Ta, Tb, and Tc in the workspace S from information from the vision sensor 11. The detection of the three-dimensional positions of reference points Ta, Tb, and Tc uses a stereo method, but in other implementations, other three-dimensional measurement methods such as TOF (time of flight), optical projection (optical sectioning, phase shifting, spatial coding, etc.), and focusing methods can also be used.

[0030] In order to calculate at least one of the position and orientation of the mobile robot 10 relative to the workspace S, the reference points Ta, Tb, and Tc must be multiple (at least two). For example, the reference points Ta, Tb, and Tc can be three target markers set in the workspace S, but they are not limited to this. In other embodiments, they can also be known feature points existing in the workspace S, such as the corners of the machine 20.

[0031] In order to accurately calculate at least one of the position and orientation of the mobile robot 10 relative to the workspace S, a plurality of reference points Ta, Tb, and Tc should be set as far apart from each other as possible. For example, two reference points Ta and Tb are set outside the machine 20 (e.g., the right and left exterior of the machine 20), and one reference point Tc is set inside the machine 20 (e.g., the upper part of the workpiece W). However, this is not a limitation. In other embodiments, one reference point may be set outside the machine 20, and two reference points may be set inside the machine 20.

[0032] The control device 12 instructs the robot 10a to move, causing the vision sensor 11 to sequentially move to at least one of the camera position and camera posture. The vision sensor 11 sequentially captures reference points Ta, Tb, and Tc. Based on the information from the vision sensor 11, the control device 12 detects the detected three-dimensional information including the reference points Ta, Tb, and Tc in the detected three-dimensional position. Based on the detected three-dimensional information and the reference three-dimensional information including the reference points Ta, Tb, and Tc in the reference three-dimensional position, the control device 12 calculates the deviation amount from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S as a correction amount, and applies correction to the movement of the mobile robot 10 based on the deviation amount (correction amount).

[0033] Figure 2 is a functional block diagram of the robot system 1 in the first embodiment. The control device 12 includes a memory unit 33, a three-dimensional information detection unit 32, a correction calculation unit 30, and a control unit 31. The memory unit 33 is composed of memory such as RAM and ROM. The components other than the memory unit 33 are composed of part or all of a computer program, but are not limited to this. In other embodiments, they may also be composed of part or all of a semiconductor integrated circuit. Furthermore, in other embodiments, the components other than the control unit 31 may also be configured on an external computer device that can be connected to the control device 12 via wired or wireless means.

[0034] The memory unit 33 stores the motion program of the mobile robot 10, the correction information (so-called internal parameters and external parameters) of the vision sensor 11, the reference three-dimensional information including the reference three-dimensional position of the reference point T, and various other information such as past deviations (corrections). Furthermore, the reference point T is an abbreviation of multiple reference points Ta, Tb, and Tc.

[0035] The 3D information detection unit 32 detects 3D information, including the 3D position of the reference point T, based on information from the vision sensor 11. As mentioned earlier, the 3D information detection unit 32 uses a stereo method to detect the 3D position of the reference point T, but it is not limited to this. In other embodiments, other 3D measurement methods such as TOF, light projection, and focusing methods can also be used. The 3D information detection unit 32 sends the detected 3D information to the correction calculation unit 30.

[0036] The correction calculation unit 30 includes a deviation calculation unit 30a. The deviation calculation unit 30a may be composed of part or all of a computer program, but is not limited thereto; in other embodiments, it may also be composed of part or all of a semiconductor integrated circuit. Based on detected three-dimensional information and pre-memorized reference three-dimensional information, the deviation calculation unit 30a calculates the deviation (correction amount) from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S. The deviation calculation unit 30a sends the calculated deviation (correction amount) to the control unit 31 and stores the deviation (correction amount) as a past correction amount in the memory unit 33.

[0037] The control unit 31 applies corrections to the motion of the mobile robot 10 based on the deviation amount (correction amount). Although the control unit 31 applies corrections to the motion of the robot 10a based on the deviation amount (correction amount), in other embodiments, it can also apply corrections to the motion of the conveying device 10b based on the deviation amount (correction amount).

[0038] When the mobile robot 10 moves near other machinery 20 or other workpieces W, causing a change in the stopping position or posture of the mobile robot 10 (conveyor 10b), or when the position or posture of the machinery 20 or workpiece W, which is the object of the operation, changes, the reference point T may be outside the field of view of the vision sensor 11. Therefore, when the memory unit 33 has already memorized the past deviation amount (correction amount), firstly, the control unit 31 automatically changes at least one of the camera position and camera posture of the vision sensor 11 based on the past deviation amount (correction amount), causing the mobile robot 10 to move. After the change, the control unit 31 sends a camera command to the vision sensor 11, and the vision sensor 11 captures the reference point T according to the camera command.

[0039] Therefore, since the likelihood of the reference point T entering the field of view of the vision sensor 11 is increased, the correction processing of the mobile robot 10 can be automated without the labor of manually changing the stopping position or stopping posture of the mobile robot 10 (transfer device 10b), manually changing the position or posture of the machine 20 or workpiece W that is the object of the operation, or re-teaching the camera position or camera posture through trial and error.

[0040] After the reference point T enters the field of view of the vision sensor 11, the 3D information detection unit 32 detects the 3D information including the detected 3D position of the reference point T based on the information detected by the vision sensor 11. The correction calculation unit 30 calculates the deviation (correction amount) from at least one of the reference position and reference posture of the mobile robot 10 relative to the work space S based on the detected 3D information and the reference 3D information including the reference 3D position of the reference point T. The control unit 31 corrects the movement of the mobile robot 10 based on the deviation (correction amount).

[0041] The following describes the detailed operation of the robot system 1 in the first embodiment with reference to Figures 3-6. Figure 3 is a flowchart of the first correction process of the robot system 1 in the first embodiment. The first correction is the process of applying correction to the robot's operation when the memory unit 33 has not stored the past deviation amount (correction amount).

[0042] In the first correction, in step S1, the mobile robot 10 (transfer device 10b) is moved to a reference position or reference posture, either manually or automatically. In step S2, the control unit 31 causes the mobile robot 10 to move, so that the vision sensor 11 moves to at least one of the pre-taught camera position and camera posture.

[0043] Figure 5 is an explanatory diagram illustrating an example of the detection of three-dimensional position by the camera positions and postures C1-C6 of the vision sensor 11 and the reference points Ta, Tb, and Tc. The camera positions and postures C1-C6 include at least one of camera position and camera posture. When using the stereo method, since two images are taken at different positions for a single reference point, there are six camera positions and postures C1-C6 when there are three reference points Ta, Tb, and Tc. However, in other embodiments using the TOF method or optical sectioning method, note that there are only three camera positions and postures. The camera positions and postures C1-C6 are pre-taught.

[0044] Referring again to Figure 3, in step S3, the vision sensor 11 captures reference points Ta, Tb, and Tc at camera positions C1 to C6. When reference points Ta, Tb, and Tc are not in the field of view of the vision sensor 11, in the first correction, in order to bring reference points Ta, Tb, and Tc into the field of view of the vision sensor 11, the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) is manually changed, or the position or posture of the machine 20 or workpiece W that is the object of the operation is manually changed, or the camera position or camera posture of the vision sensor 11 is re-taught.

[0045] As shown in Figure 5, especially when using the stereo method, if the reference points Ta, Tb, and Tc fall at the edge of the field of view of the vision sensor 11, the distance between the reference points T captured in the two images will increase (i.e., parallax), thus improving the accuracy of the three-dimensional position of the reference points Ta, Tb, and Tc. Therefore, when using the stereo method, the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) can be manually changed, or the position or posture of the machine 20 or workpiece W that is the object of the operation can be manually changed, or the camera position or camera posture of the vision sensor 11 can be re-taught, with the reference point T falling at the edge of the field of view of the vision sensor 11.

[0046] Referring again to Figure 3, in step S4, the 3D information detection unit 32 detects the 3D information including the reference points Ta, Tb, and Tc, based on the information from the vision sensor 11.

[0047] The following description uses Figure 5 to illustrate an example of detecting the three-dimensional position of reference points Ta, Tb, and Tc. For ease of explanation, in camera positions C1 and C2, C3 and C4, and C5 and C6, the vision sensor 11 is arranged in an equilateral parallel configuration. That is, in camera positions C1 and C2, C3 and C4, and C5 and C6, the vision sensors 11 are spaced apart by a baseline length B, and their optical axes O are arranged parallel to each other. An image sensor of the vision sensor 11 is arranged in a plane orthogonal to the optical axis O, with the x and y directions of the image sensor aligned in the same direction.

[0048] For example, in camera position poses C1 and C2, camera position pose C1 is set as the origin of camera coordinate system C, camera position poses C1 and C2 are set as the focal positions of visual sensor 11, the focal distance of visual sensor 11 is set as f, the parallax of reference point Ta captured in the two images is set as D, and the pixel pitch of visual sensor 11 is set as 1 mm (= 1 pixel). The distance from the origin of camera coordinate system C to reference point Ta, i.e., the Z coordinate c1z a of reference point Ta in camera coordinate system C, is calculated by the following formula using the stereo method.

[0049] [Mathematical Expression 1]

[0050] Furthermore, in c1z a, the superscript indicates the origin of the coordinate system, and the subscript indicates a coordinate point in the coordinate system. That is, c1z a means that the origin of the camera coordinate system C is located at the camera position C1, and the Z coordinate of the reference point Ta in the camera coordinate system C is...

[0051] Since the baseline length B and the focal distance f are constants determined by the pose of the two camera positions and the design of the visual sensor 11, as long as the three-dimensional information detection unit 32 calculates the parallax D of the reference point Ta in the two captured images based on image processing such as detection processing and matching processing of the reference point Ta, the z coordinates c1z a of the reference point Ta can be detected.

[0052] Furthermore, in an image coordinate system where the upper left corner of the captured image is set as the origin, if the image coordinates of the reference point T captured in the captured image are set as (x, y) and the image center of the image coordinate system is set as (cx, cy), then the three-dimensional information detection unit 32 can calculate the x coordinates c1x a and y coordinates c1y a of the reference point Ta in the camera coordinate system C, for example, using the following formula.

[0053] [Mathematical Expression 2]

[0054] [Mathematical Expression 3]

[0055] Furthermore, in c1x a and c1y a, the superscript indicates the origin of the coordinate system, and the subscript indicates a coordinate point in the coordinate system (the same applies below). That is, c1x a and c1y a respectively represent the X and Y coordinates of the reference point Ta in the camera coordinate system C when the origin of the camera coordinate system C is located at the camera position and posture C1.

[0056] Furthermore, the 3D information detection unit 32 can also perform aberration correction as needed. The internal parameters of the vision sensor 11, such as the aberration correction coefficient, focal distance f, image center (cx, cy), and pixel pitch, are preset and stored in the memory unit 33.

[0057] From the above, the detection 3D position (c1x a, c1y a, c1z a) of the reference point Ta in the camera coordinate system C is obtained. Similarly, the detection 3D positions (c3x b, c3y b, c3z b) and (c5x c, c5y c, c5z c) of the reference points Tb and Tc in the camera coordinate system C are also obtained.

[0058] Next, the 3D information detection unit 32 converts the detected 3D position of reference points Ta, Tb, and Tc from the camera coordinate system C to the mechanical coordinate system M. In the camera position and pose C1, C3, and C5, if the position and pose of the vision sensor 11 in the mechanical coordinate system M (i.e., the position and pose of the camera coordinate system C) are set as C1(mx c1, my c1, mz c1, mw c1, mp c1, mr c1), C3(mx c3, my c3, mz c3, mw c3, mp c3, mr c3), and C5(mx c5, my c5, mz c5, mw c5, mp c5, mr c5), then the external parameters (R,t)c1, (R,t)c3, and (R,t)c5 of the vision sensor 11 used to transform the three-dimensional position of the mechanical coordinate system M into the three-dimensional position of the camera coordinate system C are, for example, homogeneous transformation matrices, as shown in the following formula. Furthermore, the external parameters of the vision sensor 11 are set to be pre-memorized in the memory unit 33.

[0059] [Mathematical Expression 4]

[0060] In the above formula, R represents the rotation matrix (from the first row of the first column to the third row of the third column of the homogeneous transformation matrix), and t represents the parallel translation amount, i.e., the translation vector (from the fourth row of the first column to the fourth row of the third column of the homogeneous transformation matrix). Also, mx c1, my c1, and mz c1 are the X, Y, and Z coordinates (position of camera coordinate system C) of the camera position and orientation C1 of the vision sensor 11 in the mechanical coordinate system M, respectively. mw c1, mp c1, and mr c1 are the rotation amounts around the X-axis, Y-axis, and Z-axis (orientation of the camera coordinate system) of the camera position and orientation C1 of the vision sensor 11 in the mechanical coordinate system M, respectively. The same applies to camera position and orientation C3 and C5.

[0061] Therefore, the detected three-dimensional positions (c1x a, c1y a, c1z a), (c3x b, c3y b, c3z b), (c5x c, c5y c, c5z c) of the reference points Ta, Tb, and Tc in the camera coordinate system C are respectively converted into the three-dimensional positions (mx a, my a, mz a), (mx b, my b, mz b), and (mx c, my c, mz c) of the reference points Ta, Tb, and Tc in the mechanical coordinate system M by the following formula.

[0062] [Mathematical Expression 5]

[0063] In the above formula, RT represents the transpose of the rotation matrix R. As described above, the three-dimensional information detection unit 32 detects the three-dimensional positions of the reference points Ta, Tb, and Tc in the mechanical coordinate system M.

[0064] Referring again to Figure 3, in step S5, the deviation calculation unit 30a calculates the deviation (correction amount) from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S, based on the detection three-dimensional information including the detection three-dimensional position of reference points Ta, Tb, and Tc, and the reference three-dimensional information including the reference three-dimensional position of reference points Ta, Tb, and Tc. Furthermore, the deviation calculation unit 30a stores the calculated deviation (correction amount) as a past correction amount in the memory unit 33.

[0065] The following describes an example of calculating the deviation of the mobile robot 10 relative to at least one of its reference position and reference posture in the workspace S. Figure 6 is an explanatory diagram of an example of calculating the deviation (correction) of the mobile robot 10. Figure 6 shows the reference three-dimensional information and the detection three-dimensional information. The reference three-dimensional information includes the reference three-dimensional positions of reference points Ta, Tb, and Tc in the mechanical coordinate system M(0,0,0,0,0,0), namely Ta(mx ra, my ra, mz ra), Tb(mx rb, my rb, mz rb), and Tc(mx rc, my rc, mz rc). On the other hand, the detection of three-dimensional information includes the detection of the reference points Ta, Tb, and Tc in the mechanical coordinate system M'(0,0,0,0,0,0) that deviate from the reference three-dimensional information's mechanical coordinate system M due to deviations in the stopping position or stopping posture of the mobile robot 10 (transfer device 10b), or deviations in the position or posture of the machine 20 or workpiece W as the object of operation.

[0066] If, in the reference 3D information, a triangular pyramid is to be formed with reference points Ta, Tb, and Tc as its base and the origin of the mechanical coordinate system M as its vertex, and this pyramid rotates and moves parallel to the mechanical coordinate system M' in the detection 3D information, then the deviation (correction) calculated from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S is equivalent to the position and posture of the mechanical coordinate system M in the reference 3D information under the mechanical coordinate system M' of the detection 3D information. That is, the deviation (correction) calculated from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S can be expressed as M(m'x m, m'y m, m'z m, m'w m, m'p m, m'r m). Furthermore, m'x m, m'y m, and m'z m are the X, Y, and Z coordinates of the position of the reference 3D mechanical coordinate system M under the detection 3D information mechanical coordinate system M', respectively, and m'w m, m'p m, and m'r m are the orientation (rotation around the X-axis, rotation around the Y-axis, and rotation around the Z-axis) of the reference 3D mechanical coordinate system M under the detection 3D information mechanical coordinate system M'.

[0067] At this point, if we want the points of the triangular pyramid in the reference 3D information, namely M(0,0,0), Ta(mx ra, my ra, mz ra), Tb(mx rb, my rb, mz rb), Tc(mx rc, my rc, mz rc), to be rotated and paralleled in the mechanical coordinate system M' of the detection 3D information, and then moved to M(m'x m, m'y m, m'z m), Ta(m'x a, m'y a, m'z a), Tb(m'x b, m'y b, m'z b), Tc(m'x c, m'y c, m'z c), then the following relationship exists between the reference 3D positions of reference points Ta, Tb, and Tc and the detection 3D positions of reference points Ta, Tb, and Tc.

[0068] [Mathematical Expression 6]

[0069] In the above formula, r11~r33 represent the elements of the rotation matrix (from the first row of the first column to the third row of the third column of the homogeneous transformation matrix), and m'x m, m'y m, m'z m represent the elements of the parallel translation amount, i.e., the translation vector (from the fourth row of the first column to the fourth row of the third column of the homogeneous transformation matrix).

[0070] If the above equation is expressed as X'=T･X, multiplying both sides by the inverse matrix X⁻¹ of X, we get T=X'･X⁻¹. The inverse matrix X⁻¹ of X is obtained by adjoint matrix or elimination method. In the homogeneous transformation matrix T=X'･X⁻¹, the unknown variables are six of M(m'x m, m'y m, m'z m, m'w m, m'p m, m'r m) as deviations (corrections). Therefore, by establishing and solving at least six simultaneous equations, the deviation (correction) calculated from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S is obtained.

[0071] The above method for calculating the deviation (correction) is just one example, based on linear algebra. However, it should be noted that it is not limited to this; in other implementations, geometric or other methods can also be used. For example, since the angle or volume of the sides of the triangular pyramid in the reference 3D information is constant, the inner product of TaTb･TaM, TaTc･TaM, TbTa･TbM, TbTc･TbM, TcTa･TcM, TcTb･TcM, etc., or the volume of the triangular pyramid calculated by 1 / 6×(TaTb×TaTc)･TaM, is also constant. Therefore, at least six equations can be established: the value calculated by the mechanical coordinate system M of the reference 3D information equals the value calculated by the mechanical coordinate system M' of the detection 3D information. By solving these six simultaneous equations, the deviation (correction) m'x m, m'y m, m'z m, m'w m, m'p m, and m'r m can be obtained.

[0072] Referring again to Figure 3, in step S6, the control unit 31 applies correction to the motion of the mobile robot 10 based on the deviation amount (correction amount). That is, the control unit 31 corrects each teaching point constituting the motion trajectory of the robot 10a based on M (m'x m, m'y m, m'z m, m'w m, m'p m, m'r m) as the deviation amount (correction amount). For example, based on the following formula, each teaching point constituting the motion trajectory of the robot 10a is corrected (coordinate transformed) from the mechanical coordinate system M at the time of teaching to the mechanical coordinate system M' after the mobile robot 10 deviates from at least one of the position and posture of the robot 10 relative to the workspace S.

[0073] [Mathematical Expression 7]

[0074] As described above, the first correction is completed. After the first correction is completed, when the mobile robot 10 moves to the vicinity of other machines 20 or other workpieces W, causing the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) to change again, or when the position or posture of the machine 20 or workpiece W that is the object of the operation in the workspace S changes again, the robot system 1 corrects the movement of the mobile robot 10 according to the correction flowchart for the next and subsequent corrections.

[0075] Figure 4 is a flowchart of the next and subsequent correction process of the robot system 1 in the first embodiment. The so-called next and subsequent correction refers to the process of applying correction to the robot's actions when the memory unit 33 has already stored the past deviation amount (correction amount). First, in step S7, the mobile robot 10 (transfer device 10b) is moved manually or automatically to the vicinity of other machinery 20 or other workpieces W. In step S8, the control unit 31 automatically changes the camera position posture C1~C6 of the vision sensor 11 based on the past deviation amount (correction amount), causing the mobile robot 10 to move. That is, the camera position posture C1~C6 is corrected (transformed) into the camera position posture C1'~C6' based on M (m'x m, m'y m, m'z m, m'w m, m'p m, m'r m) as the past deviation amount (correction amount) by, for example, the following formula.

[0076] [Mathematical Expression 8]

[0077] In step S9, the vision sensor 11 captures reference points Ta, Tb, and Tc with the changed camera position posture C1'~C6'. At this time, since the camera position posture C1~C6 is corrected to camera position posture C1'~C6' based on the past deviation (correction amount), the likelihood of reference points Ta~Tc entering the field of view of the vision sensor 11 increases. That is, in the next and subsequent corrections, compared with the first correction, there is no need to manually change the stopping position or stopping posture of the mobile robot 10 (transfer device 10b), manually change the position or posture of the machine 20 or workpiece W that is the object of the operation, or re-teach the camera position or camera posture through trial and error in order to make the reference points Ta~Tc enter the field of view of the vision sensor 11. The correction process of the mobile robot 10 can be automated.

[0078] Especially when using stereoscopic methods, although the camera position postures C1 to C6 are taught by placing reference points Ta, Tb, and Tc at the edges of the two captured images (i.e., by using a longer parallax D), as shown in Figure 5, thereby improving the three-dimensional position accuracy of reference points Ta, Tb, and Tc, reference points Ta, Tb, and Tc sometimes exceed the field of view of the vision sensor 11 when the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) changes, or when the position or posture of the machine 20 or workpiece W, which is the object of the operation, changes. Therefore, by automatically correcting to the camera position postures C1' to C6' using the past deviation amount (correction amount), reference points Ta to Tc will not exceed the field of view of the vision sensor 11, and thus, the possibility of capturing images at the edge of the field of view of the vision sensor 11 increases. Therefore, even when using stereoscopic methods, corrections can be automatically applied to the movements of the mobile robot 10, and the user does not need to perform particularly difficult teaching. Additionally, it can also achieve secondary effects such as reducing the difficulty of creating the motion program for robot 10a and shortening the creation time of the motion program.

[0079] Furthermore, the processing from step S10 to step S12 in Figure 4 is the same as the processing from step S4 to step S6 in Figure 3, so the explanation is omitted.

[0080] As described above, according to the robot system 1 of the first embodiment, even when the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) changes, or when the position or posture of the machine 20 or workpiece W that is the object of the operation changes, the camera position and camera posture of the vision sensor 11 are automatically changed based on the past deviation (correction amount). Therefore, the robot correction process can be automatically performed without the labor or trial and error of manually changing the stopping position or stopping posture of the mobile robot 10 (transfer device 10b), or manually changing the position or posture of the machine 20 or workpiece W that is the object of the operation, or re-teaching the camera position or camera posture in order to bring the reference points Ta, Tb, Tc into the field of view of the vision sensor 11.

[0081] The second embodiment of the robot system 1 will now be described. Referring again to FIG1, the second embodiment of the robot system 1 calculates the movement amount of the mobile robot 10 as a correction amount by making the detection image obtained from the vision sensor 11 approximate a reference image. Based on the movement amount (correction amount), it changes at least one of the camera position and camera posture of the vision sensor 11, and applies correction to the movement of the mobile robot 10 based on the information from the changed vision sensor 11.

[0082] For ease of explanation, the amount of movement (correction) of the mobile robot 10 is defined as the amount of movement of at least one of the camera position and camera posture of the vision sensor 11 (the amount of movement of the camera coordinate system C). The amount of movement (correction) can be expressed, for example, as the position and posture C (cx c', cy c', cz c', cw c', cp c', cr c') of the camera coordinate system C before movement and after movement.

[0083] In other embodiments, the amount of movement (correction) of the mobile robot 10 may be the amount of movement of at least one of the position and orientation of the front end (e.g., flange center) of the robot 10a (the amount of movement of the flange coordinate system), or it may be the amount of movement of at least one of the position and orientation of the tool 10c (tool center point) (the amount of movement of the tool coordinate system).

[0084] Referring again to Figure 2, the difference between the robot system 1 in the second embodiment and the robot system 1 in the first embodiment is that the correction calculation unit 30 includes a movement calculation unit 30b. The movement calculation unit 30b is composed of part or all of a computer program, but is not limited to this. In other embodiments, it may also be composed of part or all of a semiconductor integrated circuit.

[0085] After the mobile robot 10 (transfer device 10b) is moved to another stop position or stop posture manually or automatically, or after the machine 20 or workpiece W, which is the object of the operation, is moved to another position or posture manually or automatically, the control unit 31 causes the mobile robot 10 to move, causing the vision sensor 11 to move to the pre-taught camera position postures C1 to C6. After the movement, the control unit 31 sends a camera command to the vision sensor 11, and the vision sensor 11 captures the reference point T according to the camera position postures C1 to C6.

[0086] The motion calculation unit 30b determines whether the reference point T appears in the detection image obtained from the vision sensor 11. When the reference point T does not appear in the detection image, the motion calculation unit 30b sends an arbitrary motion amount (correction amount) to the control unit 31. The control unit 31 changes at least one of the camera position and camera posture of the vision sensor 11 based on the arbitrary motion amount (correction amount), causing the mobile robot 10 to move. The arbitrary motion amount (correction amount) can be a pre-specified motion amount (correction amount), for example, represented as C(cx sc', cy sc', cz sc', cw sc', cp sc', cr sc'). For example, the arbitrary motion amount (correction amount) includes four motion amounts (correction amounts) that sequentially move the camera position horizontally by a predetermined amount in the up, down, left, and right directions before the change, four motion amounts (correction amounts) that sequentially rotate the camera posture horizontally by a predetermined amount in the up, down, left, and right directions before the change, or four motion amounts (correction amounts) that combine these. Furthermore, in other embodiments, any correction amount can also be the deviation amount in the first embodiment. After the change, the control unit 31 sends the camera command to the vision sensor 11, and the vision sensor 11 captures the reference point T according to the camera command.

[0087] On the other hand, when the reference point T appears in the detection image, the movement calculation unit 30b calculates the movement amount (correction amount) in such a way that the detection image obtained from the vision sensor 11 is close to the reference image pre-memorized in the memory unit 33. The movement amount (correction amount) is calculated using machine learning, which will be described later.

[0088] Next, the motion calculation unit 30b determines whether the calculated motion (correction) is below a threshold. When the motion (correction) exceeds the threshold, since the reference point T is more likely not at the edge of the field of view of the vision sensor 11, the motion calculation unit 30b sends the calculated motion (correction) to the control unit 31. The control unit 31 changes at least one of the camera position and camera posture of the vision sensor 11 based on the calculated motion (correction), causing the mobile robot 10 to move. After the change, the control unit 31 sends a camera command to the vision sensor 11, and the vision sensor 11 captures the reference point T according to the camera command.

[0089] On the other hand, when the amount of movement (correction) is below the threshold, since the reference point T is more likely to be at the edge of the field of view of the vision sensor 11, the amount of movement calculation unit 30b sends the calculation instruction of the deviation amount (correction) to the deviation amount calculation unit 30a. The deviation amount calculation unit 30a calculates the deviation amount (correction) from at least one of the reference position and reference posture of the mobile robot 10 relative to the work space S, as described in the first embodiment.

[0090] The deviation calculation unit 30a sends the calculated deviation (correction amount) to the control unit 31, which applies correction to the movement of the mobile robot 10 based on the deviation (correction amount). Although the control unit 31 applies correction to the movement of the robot 10a based on the deviation (correction amount), in other embodiments where the conveying device 10b is an automated conveying device, the control unit 31 can also apply correction to the movement of the conveying device 10b based on the deviation (correction amount).

[0091] The detailed operation of the robot system 1 in the second embodiment will be described below with reference to Figures 7 and 8. Figure 7 is a correction flowchart of the robot system 1 in the second embodiment. First, in step S1, the mobile robot 10 (transfer device 10b) is moved manually or automatically to the vicinity of other machinery 20 or other workpieces W. In step S2, the control unit 31 causes the mobile robot 10 to move, moving the vision sensor 11 to the pre-taught camera position postures C1 to C6 (see Figure 5).

[0092] In step S3, the vision sensor 11 captures reference points Ta, Tb, and Tc in camera position postures C1 to C6. In step S4, the motion calculation unit 30b determines whether reference points Ta, Tb, and Tc appear in the detection image obtained from the vision sensor 11. The detection of reference points Ta, Tb, and Tc can be performed using image processing such as matching processing. When reference points Ta, Tb, and Tc do not appear in the detection image (no in step S4), the control unit 31 automatically changes the camera position postures C1 to C6 of the vision sensor 11 based on arbitrary motion amounts (correction amounts), causing the mobile robot 10 to move. That is, the camera position postures C1 to C6 are corrected (converted) to camera position postures C1' to C6', for example, using the following formula, based on C (cx sc', cy sc', cz sc', cw sc', cp sc', cr sc') as arbitrary motion amounts (correction amounts).

[0093] [Mathematical Expression 9]

[0094] Then, returning to step S3, the vision sensor 11 re-captures reference points Ta, Tb, and Tc in the changed camera position posture C1'~C6'. In step S4, the motion calculation unit 30b again determines whether reference points Ta, Tb, and Tc appear in the detection image obtained from the vision sensor 11. When reference points Ta, Tb, and Tc appear in the detection image (in step S4), the motion calculation unit 30b calculates the motion amount (correction amount) in a way that makes the detection image obtained from the vision sensor 11 approximate the reference image pre-memorized in the memory unit 33.

[0095] The following explanation, using FIG8, describes the machine learning used to calculate the movement amount (correction amount). The diagram illustrates the calculation principle of the robot's movement amount (correction amount) in the second embodiment. The movement amount calculation unit 30b observes the detection image obtained from the vision sensor 11 and the movement amount (correction amount) of the mobile robot 10 calculated by making the detection image approximate a reference image as state variables. Furthermore, the movement amount calculation unit 30b obtains the reference image from the memory unit 33 as decision data.

[0096] Next, the motion calculation unit 30b learns the amount of motion (correction) that enables the mobile robot 10 to move from at least one of any position and posture to at least one of a reference position and a reference posture based on a training data set consisting of a combination of state variables and decision data.

[0097] The motion calculation unit 30b can calculate a reward based on at least one of the camera position and camera posture of the visual sensor 11 after movement, and the reference position of the reference point T. Based on the reward, it updates the function used to infer the motion (correction) of the visual sensor 11 from the current state variables. That is, the motion calculation unit 30b can use so-called Q-learning for reinforcement learning.

[0098] Alternatively, the motion calculation unit 30b can use the detection image obtained from the visual sensor 11, which has moved to at least one of the predetermined positions and postures, and the motion amount (correction amount) of the visual sensor 11 moving from at least one of the predetermined positions and postures towards the reference position of the reference point T as a label for supervised learning. By performing supervised learning, reference images taken at least one of the reference positions and postures are obtained, and the motion amount (correction amount) and the detection image taken at least one of the reference positions and postures are memorized after moving to an appropriate position. A plurality of combinations of reference images and detection images are prepared, thereby learning the relationship between image changes and motion amount (correction amount), and automatically obtaining a large amount of learning data.

[0099] The control unit 31 can also move the vision sensor 11 based on the amount of movement (correction) calculated in a way that makes the detection image obtained from the vision sensor 11 approach the reference image. The closer the position and posture of the vision sensor 11 after the movement are to the reference position and reference posture, the higher the reward is given.

[0100] The movement calculation unit 30b should update the action value table corresponding to the movement (correction) of the visual sensor 11 based on the state variables and rewards. The movement calculation unit 30b should also use a multi-layer structure to calculate the observed state variables and update the action value table used to determine the action value in real time. Here, the method of calculating the state variables using a multi-layer structure can use a so-called multi-layer neural network.

[0101] The motion calculation unit 30b can also update the value function corresponding to the motion (correction) of the visual sensors of other mobile robots with the same structure as the mobile robot 10, based on the state variables and rewards of those robots. That is, it can update its own value function not by using the value function updated by the motion calculation unit 30b, but by using the value function updated by other machine learning devices. For example, it can also send and receive data among multiple control devices 12, and utilize the learning content of other control devices 12 for its own learning.

[0102] The movement calculation unit 30b can also be configured to update the movement (correction) of the mobile robot 10 by learning from an additional training dataset consisting of a combination of current state variables and decision data.

[0103] The motion calculation unit 30b shall determine the motion command for the mobile robot 10 based on the results learned from the training dataset. The motion calculation unit 30b uses the detection image captured by at least one of the camera position and camera posture after the vision sensor 11 has moved as a state variable, and the reference image as the decision data, to calculate the motion amount (motion data) of the mobile robot 10 that makes the reference point T appearing in the vision sensor 11 approach the reference point T in the reference image. For example, the motion calculation unit 30b calculates the motion amount (correction amount) of the vision sensor 11 in such a way that the detection image obtained from the vision sensor 11 moved to an arbitrary position approaches the reference image.

[0104] Referring again to Figure 7, in step S7, the motion calculation unit 30b determines whether the calculated motion (correction) is below a threshold. When the motion (correction) exceeds the threshold (No in step S7), since the reference points Ta, Tb, and Tc are more likely not at the edge of the field of view of the vision sensor 11, in step S8, the control unit 31 changes the camera position posture C1 to C6 of the vision sensor 11 based on the calculated motion (correction), causing the mobile robot 10 to move. Then, returning to step S3, the vision sensor 11 takes another picture of the reference points Ta, Tb, and Tc with the changed camera position posture C1' to C6'. In step S4, the motion calculation unit 30b determines again whether the reference points Ta, Tb, and Tc appear in the detection image obtained from the vision sensor 11. When reference points Ta, Tb, and Tc appear in the detection image (as in step S4), the movement calculation unit 30b recalculates the movement amount (correction amount) in such a way that the detection image obtained from the vision sensor 11 is close to the reference image.

[0105] On the other hand, when the amount of movement (correction) is below a threshold (as in step S7), since the reference points Ta, Tb, and Tc are more likely to be at the edge of the field of view of the vision sensor 11, the deviation calculation unit 30a in step S9 calculates the deviation (correction) from at least one of the reference position and reference posture of the mobile robot 10 relative to the workspace S, as described in the first embodiment. Then, in step S10, the control unit 31 applies correction to the movement of the mobile robot 10 based on the deviation (correction). Although the control unit 31 applies correction to the movement of the robot 10a based on the deviation (correction), in other embodiments where the conveying device 10b is an automatic conveying device, the movement of the conveying device 10b can also be corrected based on the deviation (correction).

[0106] As described above, according to the robot system 1 of the second embodiment, even when the stopping position or stopping posture of the mobile robot 10 (transfer device 10b) changes, or when the position or posture of the machine 20 or workpiece W that is the object of the operation changes, the mobile robot 10 is moved based on the movement amount (correction amount) calculated by machine learning, and the movement and imaging of the vision sensor 11 are repeated. Therefore, in order to make the reference points Ta, Tb, Tc fall at the edge of the field of view of the vision sensor 11, the robot can automatically perform correction processing by manually changing the stopping position or stopping posture of the mobile robot (transfer device 10b), or manually changing the position or posture of the machine 20 or workpiece W that is the object of the operation, or re-teaching the imaging position or imaging posture.

[0107] Furthermore, according to the robot system 1 of the second embodiment, when determining whether reference points Ta, Tb, and Tc appear in the detection image, if reference points Ta, Tb, and Tc do not appear in the detection image, the camera position and posture C1 to C6 of the vision sensor 11 will be automatically changed based on any amount of movement (correction amount). Therefore, there is no need for manual changes to the stopping position or stopping posture of the mobile robot (transfer device 10b), or manual changes to the position or posture of the machine 20 or workpiece W that is the object of the operation, or re-teaching the camera position or posture in order to bring reference points Ta, Tb, and Tc into the field of view of the vision sensor 11. The robot's correction processing can be automated.

[0108] The aforementioned computer program may be provided by recording on a computer-readable non-transitory recording medium such as a CD-ROM, or by transmitting via wired or wireless means from a server device on a WAN (wide area network) or LAN (local area network).

[0109] Various embodiments have been described in this specification, but the present invention is not limited to the aforementioned embodiments. It should be understood that various modifications can be made within the scope of the following claims.

[0110] 1: Robot System 10: Mobile Robots 10a: Robot 10b:Conveying device 10c: Tools 11: Vision Sensor 12: Control device 13: Teaching device 20: Machinery 21: Tools 22: Control device 30: Correction Quantity Calculation Section 30a: Deviation Calculation Section 30b: Calculation of movement quantity 31: Control Department 32: 3D Information Inspection Department 33: Memory Department B: Baseline length C,C': Camera coordinate system C1~C6, C1'~C6': Camera position, camera position posture c1z a, c3z b, c5z c: Z coordinate D: Parallax f: Focal distance M,M': Mechanical coordinate system O: Optical axis S: Workspace S1~S12: Steps T,Ta,Tb,Tc: Reference points W: Workpiece

Claims

1. A control device comprising: a correction amount calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from a vision sensor mounted on a mobile robot; and a control unit that changes at least one of a camera position and a camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the correction amount calculation unit calculates a deviation amount calculated from at least one of a reference position and a reference posture of the robot relative to the workspace as the correction amount based on detection three-dimensional information and reference three-dimensional information detected according to the information from the vision sensor, and the control unit changes at least one of the camera position and the camera posture of the vision sensor based on the deviation amount.

2. The control device of claim 1, wherein the aforementioned control unit changes at least one of the aforementioned camera position and the aforementioned camera posture of the aforementioned visual sensor based on the aforementioned past correction amount.

3. A control device comprising: a correction calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from a vision sensor mounted on a mobile robot; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the control unit changes at least one of the camera position and camera posture of the vision sensor based on an arbitrary correction amount when a reference point of the workspace does not appear in a detection image obtained from the vision sensor.

4. A control device comprising: a correction amount calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from a vision sensor mounted on a mobile robot; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the correction amount calculation unit calculates the robot's movement as the correction amount by making the detection image approach a reference image when a reference point in the workspace appears in a detection image obtained from the vision sensor, and the control unit changes at least one of the camera position and camera posture of the vision sensor based on the movement amount.

5. The control device according to any one of claims 1 to 4, wherein the aforementioned correction calculation unit calculates the aforementioned correction amount using machine learning, and the aforementioned control unit repeats the movement and imaging of the aforementioned visual sensor based on the aforementioned correction amount.

6. A robot system comprising: a mobile robot; a vision sensor mounted on the mobile robot; a correction calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from the vision sensor; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the correction calculation unit calculates a deviation amount calculated from at least one of the reference position and reference posture of the robot relative to the workspace as the correction amount based on detection three-dimensional information and reference three-dimensional information detected according to the information from the vision sensor, and the control unit changes at least one of the camera position and camera posture of the vision sensor based on the deviation amount.

7. A robot system comprising: a mobile robot; a vision sensor mounted on the mobile robot; a correction calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from the vision sensor; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the control unit changes at least one of the camera position and camera posture of the vision sensor based on an arbitrary correction amount when a reference point of the workspace does not appear in a detection image obtained from the vision sensor.

8. A robot system comprising: a mobile robot; a vision sensor mounted on the mobile robot; a correction calculation unit that calculates a correction amount for the robot's movement relative to a workspace from information from the vision sensor; and a control unit that changes at least one of the camera position and camera posture of the vision sensor based on the correction amount, and applies correction to the robot's movement based on the changed information from the vision sensor, wherein the correction calculation unit calculates the robot's movement as the correction amount by making the detection image approach the reference image when a reference point in the workspace appears in a detection image obtained from the vision sensor, and the control unit changes at least one of the camera position and camera posture of the vision sensor based on the movement amount.

Images