Correction system, information processing system, robot control system, correction method, information processing method, robot control method, correction device, information processing device, and robot control device

Abstract

A correction device of one aspect of the present invention generates first coordinate information indicating a position of a gripping portion in a photographing image coordinate system that is based on a photographing image obtained by photographing the gripping portion from a prescribed direction, based on a learning model generated by machine learning on a plurality of teaching data and the photographing image, in a correction system that includes a robot, a photographing device, and the correction device, wherein the robot includes the gripping portion that grips a gripping object at a distal end thereof, and the plurality of teaching data has an image obtained by capturing a virtual space including a three-dimensional model related to the gripping portion, and information indicating a position of the gripping portion included in the virtual space. The correction device coordinates-converts the first coordinate information into second coordinate information using coordinate conversion information used in coordinate conversion processing that coordinates-converts the first coordinate information into the second coordinate information, wherein the second coordinate information indicates a position of the gripping portion in a robot coordinate system that is based on the robot.

Description

TECHNICAL FIELD

[0001] The present application relates to a correction system, an information processing system, a robot control system, a correction method, an information processing method, a robot control method, a correction device, an information processing device, and a robot control device. BACKGROUND

[0002] A technique is known in which a three-dimensional position of a robot arm is recognized so as to grip a plurality of objects (workpieces) in bulk with the robot arm or the like. A technique is known in which, at this time, teaching data including an image obtained by photographing the robot arm is used to generate a learning model for estimating a position of the robot arm from the image. When adjusting a positional relationship of the robot arm and a camera, a marker such as a two-dimensional code, a color pattern, or the like that becomes a recognition target of the camera is sometimes attached to the robot arm, and the marker is detected by the camera. In this case, a position of the marker detected by photographing with the camera is used as the teaching data in the learning process.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Laid-Open (JP A) No. 2018-144152

[0006] Patent Literature 2: Japanese Patent Application Laid-Open (JP A) No. 2019-171540 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] However, it is difficult to attach the marker to a position of a hand of the robot arm that becomes a detection target, that is, a gripping portion, and thus adjustment of the positional relationship is sometimes not accurately performed. In order to accurately perform adjustment of the positional relationship, work such as replacement of the gripping portion of the robot arm with a dedicated jig or the like is required.

[0009] The present application is made in view of the above-described problem, and aims to provide a correction system, an information processing system, a robot control system, a correction method, an information processing method, a robot control method, a correction device, an information processing device, and a robot control device that can easily adjust a positional relationship of a robot arm and a camera.

[0010] SOLUTION TO THE PROBLEM

[0011] The correction system of one aspect of the present application includes a robot arm, an imaging device, and a correction device. The robot arm includes a gripping portion at a distal end thereof that grips an object. The imaging device generates at least one captured image by capturing the robot arm from a predetermined direction. The correction device includes a first coordinate generation portion, a second coordinate generation portion, and a conversion portion. The first coordinate generation portion generates first coordinate information indicating a position of the gripping portion in a captured image coordinate system based on a learning model generated by machine learning of a plurality of teaching data and the captured image captured from the predetermined direction, the plurality of teaching data including an image captured from a virtual space including a three-dimensional model related to the robot arm and information indicating a position of the gripping portion of the robot arm in the virtual space. The conversion portion converts the first coordinate information into second coordinate information indicating a position of the gripping portion in a robot coordinate system based on coordinate conversion information used in coordinate conversion processing of the first coordinate information into the second coordinate information.

[0012] Effects of Invention

[0013] According to one aspect of the present application, the positional relationship between the robot arm and the camera can be easily adjusted. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a diagram showing one example of a correction system of the first embodiment.

[0015] Figure 2 FIG. 2 is a diagram showing one example of the relationship between a robot coordinate system and a camera coordinate system.

[0016] Figure 3 FIG. 3 is a block diagram showing one example of the structure of the correction system of the first embodiment.

[0017] Figure 4 FIG. 4 is a diagram showing one example of a three-dimensional model of the gripping portion.

[0018] Figure 5 FIG. 5 is a diagram showing one example of a captured image of a virtual space in which the three-dimensional model of the gripping portion is arranged.

[0019] Figure 6 FIG. 6 is a flowchart showing one example of the learning processing of the first embodiment.

[0020] Figure 7 FIG. 7 is a flowchart showing one example of the correction processing of the first embodiment.

[0021] Figure 8 FIG. 8 is a block diagram showing one example of the structure of the correction system of the second embodiment.

[0022] Figure 9 is a flowchart showing one example of the correction process of the second embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, the correction system, the information processing system, the robot control system, the correction method, the information processing method, the robot control method, the correction device, the information processing device, and the robot control device of the embodiments will be described with reference to the drawings. Note that the present application is not limited to the embodiments. In addition, the dimensional relationship of each element, the ratio of each element, and the like in the drawings can sometimes be different from the actual. Sometimes the drawings can include parts different from each other in the dimensional relationship and the ratio. In addition, the contents described in one embodiment or modified example are also basically applicable to other embodiments or modified examples.

[0024] (First Embodiment)

[0025] Figure 1 is a diagram showing one example of the correction system of the first embodiment. Figure 1 The correction system 1 illustrated in FIG. 1 is provided with a photographing device 20, a robot arm 30, and a correction device 10 not illustrated which will be described later. The correction process is performed, for example, at the timing of setting the photographing device 20 or the timing of starting the robot arm 30.

[0026] The photographing device 20 photographs an image including the gripping portion 31 of the robot arm 30 and outputs it to the correction device 10. In the photographing device 20 of the first embodiment, for example, a camera capable of photographing a plurality of images such as a known stereo camera can be used. The photographing device 20 is provided at a position where both the gripping portion 31 of the robot arm 30 and the workpieces 41, 42, and the like can be photographed. As will be described later, when the correction process is performed, the photographing device 20 photographs the work area including the gripping portion 31 of the robot arm 30 a predetermined number of times (n times). Note that in the first embodiment, the distortion of the individual lens of the photographing device 20, the normalization of the brightness, the parallelization between the plurality of lenses (stereo combination) included in the photographing device 20, and the like have been corrected respectively.

[0027] The correction device 10 uses the image output from the imaging device 20 to estimate the position of the robot arm 30 or the like. The correction device 10 generates a coordinate conversion matrix D, which will be described later, based on the coordinate system in the image output from the imaging device 20 and the coordinate system with reference to the joint angle of the robot arm 30, for example, in the correction process. The correction device 10 converts the coordinate system coordinates in the image output from the imaging device 20 to the coordinate system with reference to the joint angle of the robot arm 30, for example, using the generated coordinate conversion matrix D. Further, the correction device 10 outputs a signal that controls the action of the robot arm 30 based on the position of the gripper 31 of the robot arm 30 after the coordinate conversion, the position of the workpieces 41, 42, or the like.

[0028] The robot arm 30 has a gripper (gripper claw) 31 that grips an object and a plurality of joints 32a to 32f that change the position of the gripper 31. The robot arm 30 changes the joint angles of the plurality of joints 32a to 32f using a known control method based on the signal output from the correction device 10, for example. Thus, the robot arm 30 performs an action of moving the gripper 31 to grip the workpieces 41, 42, or the like. For example, as will be described later, the robot arm 30 moves the gripper 31 before the imaging device 20 images the work area at the time of the correction process. Note that the workpieces 41, 42 are one example of the object.

[0029] The position of the robot arm 30 and the gripper 31 controlled by the correction device 10, that is, the position at which the robot arm 30 and the gripper 31 actually exist, is expressed by a coordinate system (hereinafter, sometimes referred to as “robot coordinate system V”) defined by the X axis, the Y axis, and the Z axis shown in FIG. 1. Further, the position of the robot arm 30 and the gripper 31 imaged by the imaging device 20, that is, the position of the robot arm 30 and the gripper 31 on the imaged image, is expressed by a coordinate system (hereinafter, sometimes referred to as “camera coordinate system V’”) defined by the X’ axis, the Y’ axis, and the Z’ axis shown in FIG. 1. Figure 1 Figure 1

[0030] [Equation 1]

[0031]

[0032] There is sometimes a deviation between the designed positional relationship of the position of the gripper 31 of the robot arm 30 controlled by the correction device 10 and the position 31’ of the gripper 31 imaged by the imaging device 20 and the actual positional relationship at the time of the installation. This positional deviation is caused by, for example, a mechanical error between the designed value of the stereoscopic combination included in the imaging device 20 and the actual value, an error from the designed value of the distance at the time of the installation of the imaging device 20 and the robot arm 30, or the like. Figure 2 ​​is a diagram showing an example of the relationship between the robot coordinate system and the camera coordinate system. As shown in Figure 2 the position 30' of the robot arm 30 and the position 31' of the gripper 31 in the camera coordinate system V' indicated by a solid line deviate from the positions of the robot arm 30 and the gripper 31 in the robot coordinate system V indicated by a broken line.

[0033] Accordingly, the camera coordinate system V' is converted into the robot coordinate system V by the following equation (2). The coordinate conversion matrix D shown in equation (2) is, for example, a rotation-translation matrix for converting coordinates as shown in equation (3). Note that hereinafter, the coordinate conversion for adjusting the deviation of the camera coordinate system V' from the robot coordinate system V will be sometimes referred to as "correction".

[0034] V = DV'... (2)

[0035] [Equation 2]

[0036]

[0037] Note that in equation (3), t is, for example, a translation vector of three rows by one column. Further, R is, for example, a rotation matrix of three rows by three columns as shown in equation (4). That is, the coordinate conversion matrix D is represented by a four-row by four-column matrix. Note that the rotation matrix shown in equation (4) is an example of a matrix that rotates the camera coordinate system V' around the X axis.

[0038] [Equation 3]

[0039]

[0040] To calculate the coordinate conversion matrix D, first, the position 31' of the gripper 31 in the camera coordinate system V' and the position of the gripper 31 in the robot coordinate system V need to be determined. The position of the gripper 31 in the robot coordinate system V is determined, for example, by the rotation and translation of the coordinate system of each joint from the coordinate system of the other joint when viewed. On the other hand, as described above, to accurately determine the position 31' of the gripper 31 in the camera coordinate system V', a dedicated jig or the like needs to be used.

[0041] Thus, in the first embodiment, the correction system 1 uses the learning model to estimate the position 31' of the gripping portion 31 in the camera coordinate system V'. At this time, in the first embodiment, a three-dimensional model of the gripping portion 31 of the robot arm 30 is arranged on a virtual space, the position information of the three-dimensional model of the gripping portion 31 of the robot arm 30 on the virtual space is determined from the image, and thus teaching data for constructing the learning model is acquired. In the first embodiment, a combination of a captured image obtained by capturing the virtual space in which the three-dimensional model of the gripping portion 31 is arranged and the position information (coordinates) of the three-dimensional model in the captured image is used as the teaching data.

[0042] Further, in the first embodiment, the correction system 1 converts the coordinates V'i representing the position 31' of the gripping portion 31 in the camera coordinate system V' estimated using the learning model from the image captured the i-th time into coordinates Vi representing the position 31' of the gripping portion 31 in the robot coordinate system at the time of the i-th capturing. At this time, for example, the correction system 1 calculates the coordinates Vi representing the position of the gripping portion 31 in the robot coordinate system at the time of capturing the image corresponding to the coordinates V'i. The coordinates Vi are calculated, for example, based on the rotation and translation of the coordinate systems of the other joints as viewed from the coordinate system of each joint of the robot arm 30. Then, the correction system 1 estimates the coordinate conversion matrix D using, for example, the error between the coordinates V'i in the camera coordinate system V' and the coordinates Vi in the robot coordinate system V. Note that hereinafter, the rotation and translation of the coordinate systems of the other joints as viewed from the coordinate system of each joint of the robot arm 30 will be sometimes referred to as "displacement of joint angles". Further, hereinafter, the coordinates related to the position 31' of the gripping portion 31 in the camera coordinate system V' will be sometimes referred to as "first coordinate information", and the coordinates related to the position of the gripping portion 31 in the robot coordinate system V will be sometimes referred to as "second coordinate information".

[0043] Figure 3 is a block diagram illustrating one example of the structure of the correction system of the first embodiment. As shown in Figure 3 , the correction device 10 is connected to the capturing device 20 and the robot arm 30 in a manner that enables communication through the network NW. Further, as shown in Figure 3 , the correction device 10 includes a communication section 11, an input section 12, a display section 13, a storage section 14, and a processing section 15.

[0044] The communication section 11 controls the communication of data input and output to and from external devices through the network NW. For example, the communication section 11 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like, receives image data output from the capturing device 20, and transmits a signal output to the robot arm 30.

[0045] The input section 12 is connected to the processing section 15, converts an input operation received from an administrator (not shown) of the correction device 10 into an electric signal, and outputs the electric signal to the processing section 15. The input section 12 is, for example, a switch, a button, a mouse, a keyboard, a touch panel, or the like. Further, the input section 12 can also be an interface or the like for connecting an external input device to the correction device 10.

[0046] The display section 13 is connected to the processing section 15, and displays various information and various image data output from the processing section 15. The display section 13 is realized by, for example, a liquid crystal monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, or an organic EL (Electro Luminescence), or the like.

[0047] The storage section 14 is realized by, for example, a RAM (Random Access Memory) or a magnetic storage device, or the like. Various programs executed by the processing section 15 are stored in the storage section 14. Further, various formulas used when the processing section 15 executes various programs, various data output from the imaging device 20, and the like are temporarily stored in the storage section 14. The storage section 14 stores a learning model 141 and coordinate conversion information 142.

[0048] The learning model 141 is used for processing for estimating a position 31' of the gripping section 31 of the robot arm 30 in the camera coordinate system V' from an image output from the imaging device 20. The learning model 141 has, for example, a neural network structure 141a and a learning parameter 141b. The neural network structure 141a is, for example, a network structure using a publicly known network such as a convolutional neural network. The learning parameter 141b is, for example, a weight of a convolution filter of the convolutional neural network, and is a parameter to be learned and optimized for estimating the position 31' of the gripping section 31 of the robot arm in the camera coordinate system V'.

[0049] The learning model 141 is generated, for example, by learning using a combination of a captured image obtained by capturing a virtual space in which a three-dimensional model on which the gripping section 31 is arranged, and position information (coordinates) of the three-dimensional model in the captured image as teaching data. The learning model 141 is generated or updated, for example, by a learning section 152 described later.

[0050] The coordinate conversion information 142 is, for example, the coordinate conversion matrix D described above for converting the camera coordinate system V' into the robot coordinate system V. The coordinate conversion information 142 is generated, for example, by a conversion information generation section 156 described later.

[0051] The processing section 15 is realized by a processor such as a CPU (Central Processing Unit). The processing section 15 controls the entire calibration device 10. The processing section 15 performs various processes by reading various programs stored in the storage section 14 and executing the read programs. For example, the processing section 15 has a space generation section 151, a learning section 152, a robot control section 153, a first coordinate generation section 154, a second coordinate generation section 155, a conversion information generation section 156, and a conversion section 157.

[0052] The space generation section 151 generates a virtual space that contains a three-dimensional model of the gripping portion 31 of the robot arm 30. The space generation section 151 acquires a three-dimensional model of the gripping portion 31 of the robot arm 30. The three-dimensional model can be acquired, for example, by a publicly known 3D scanning or the like.

[0053] Figure 4 is a diagram showing one example of a three-dimensional model of a gripping portion. As shown in Figure 4 In the three-dimensional model of the gripping portion 31 of the robot arm 30, for example, a pair of gripping portions 31a and 31b are shown. Note that, Figure 4 The marks 33a and 33b shown by the broken lines in the three-dimensional model of the gripping portion 31a and 31b indicate the positions of the gripping portions 31a and 31b, and are not illustrated in the virtual space explained later. The position information of the marks 33a and 33b is expressed, for example, in the form of metadata attached to the generated virtual space.

[0054] Next, the space generation section 151 sets various conditions when the three-dimensional model of the gripping portion 31 is arranged on the virtual space. The number, position, posture, and the like of the arranged three-dimensional model can be set in a manner that the object is randomly generated by image generation software, but are not limited thereto, and can be arbitrarily set by the administrator of the calibration device 10.

[0055] Next, the space generation section 151 arranges the three-dimensional model on the virtual space in accordance with the set conditions. The arrangement of the three-dimensional model on the virtual space can be performed, for example, using publicly known image generation software or the like. The virtual space in which the three-dimensional model is arranged will be explained in detail later.

[0056] In order to secure the number of teaching data required for the learning process, the space generation section 151 repeatedly performs the setting of the conditions for arranging the three-dimensional model and the arrangement of the three-dimensional model on the virtual space after the capturing in the learning section 152 explained later. In this way, by acquiring and arranging the three-dimensional model on the virtual space, the position of the gripping portion 31 of the robot arm 30 arranged at an arbitrary position on the robot coordinate system V in the camera coordinate system V' can be more accurately determined on the virtual space.

[0057] Returning to Figure 3The learning unit 152 performs learning processing for generating or updating the learning model 141 using data of a virtual space in which the three-dimensional model of the gripping portion 31 of the robot arm 30 is arranged. The learning unit 152 acquires an image representing the position of the three-dimensional model of the gripping portion 31 of the robot arm 30 that has been arranged, for example, by capturing the virtual space in which the three-dimensional model of the gripping portion 31 of the robot arm 30 is arranged.

[0058] Figure 5 is an example of a captured image representing a virtual space in which a three-dimensional model of a gripping portion is arranged. As shown in Figure 5 , the three-dimensional model of the gripping portion 31 of the robot arm 30 is arranged at a position on the virtual space that meets a predetermined condition. Further, as described above, the combination of the captured image obtained by capturing the virtual space in which the three-dimensional model is arranged on the virtual space and the position information (coordinates) of the three-dimensional model in the captured image is used as teaching data in the learning processing.

[0059] Note that, as shown in Figure 5 , the three-dimensional model of a workpiece 41, 42, or the like, or a tray, or the like, on which the workpiece 41, 42 is arranged, can also be arranged on the virtual space. Further, as shown in Figure 5 , the three-dimensional model of a plurality of gripping portions can also be arranged on the virtual space. In this case, not only the position information of the gripping portion 31a, but also the position information of the gripping portion 91a, the position information of the workpieces 41, 42, the tray, or the like, are used as part of the teaching data.

[0060] Note that, in the virtual space, the three-dimensional models of a plurality of gripping portions can be arranged, for example, as a pair of left and right gripping portions 31a and 31b as shown in Figure 4 , but are not limited thereto. For example, the three-dimensional models of the gripping portions can be arranged one by one (only the gripping portion 31a). For example, the three-dimensional models of the gripping portions can be arranged in groups of three, or in groups of four. For example, in the virtual space, the three-dimensional models of a plurality of gripping portions 31a and 91a can be arranged at random positions as shown in Figure 5 . Further, the number of three-dimensional models of gripping portions arranged on the virtual space is not limited to one or two, but can be three or more.

[0061] In the first embodiment, the position of the three-dimensional model of the gripping portion 31 of the robot arm 30 is represented by three-dimensional coordinates (x, y, z), for example. Further, the learning unit 152 establishes a correspondence between the acquired image and the position, posture, or the like, of the three-dimensional model of the gripping portion 31 of the robot arm 30 that has been arranged, and stores the same in the condition storage unit 14.

[0062] To ensure the required amount of teaching data for the learning process, the learning unit 152 repeatedly performs the storage of the corresponding images and conditions for a predetermined number of times. Then, the learning unit 152 performs the learning process for a predetermined number of times using the generated teaching data, thereby generating or updating the learning parameters 141b used as weights in the neural network structure 141a.

[0063] The robot control unit 153 controls the movement of the robot arm 30 and the gripping portion 31. The robot control unit 153 controls the movement of the robot arm 30, for example, by outputting information indicating the rotation amounts of the respective joints 32a to 32f of the robot arm 30 and the opening and closing angles of the gripping portion 31.

[0064] The robot control unit 153, for example, when performing the correction process, rotates the respective joints 32a to 32f of the robot arm 30 one by one by a predetermined amount before the imaging device 20 captures an image. Thereby, the robot control unit 153 moves the position of the gripping portion 31. Then, the moved gripping portion 31 is captured by the imaging device 20. By repeatedly performing the movement and the capturing of the gripping portion 31 n times, n images of the gripping portion 31 of the robot arm 30 at different positions and n pieces of information related to the positions of the gripping portion 31 of the robot arm 30 in the robot coordinate system V are obtained.

[0065] Further, when the gripping portion 31 performs the movement of gripping the workpieces 41, 42, and the like, the robot control unit 153 generates and outputs information for controlling the robot arm 30 to the robot arm 30. At this time, the robot control unit 153 generates the information for controlling the robot arm 30 based on the coordinates indicating the positions of the workpieces 41, 42, and the like, the gripping portion 31 of the robot arm 30 in the robot coordinate system V, which are converted by the conversion unit 157. Note that the robot control unit 153 may, for example, also be a structure that moves the positions of a plurality (p) of gripping portions 31.

[0066] The first coordinate generation unit 154 generates first coordinate information as coordinate information in the camera coordinate system V’ using the images captured by the imaging device 20 and the learning model, which contain the gripping portion 31. The first coordinate generation unit 154 acquires n images captured by the imaging device 20, for example, by the communication unit 11. The first coordinate generation unit 154 generates n pieces of first coordinate information V’1, V’2, …, V’n from the acquired n images. n At this time, the first coordinate generation unit 154 may, for example, also measure the distance in the camera coordinate system V’ from the imaging device 20 based on the parallax of the left and right images captured by the imaging device 20. Note that in the case where, for example, the robot control unit 153 moves the positions of a plurality of gripping portions 31, or in the case where the images captured by the imaging device 20 contain a plurality of gripping portions 31, the first coordinate generation unit 154 generates a plurality of pieces of first coordinate information V’1, V’2, …, V’n for one image. nFor example, the first coordinate generation unit 154 generates n×p first coordinate information V′ based on n images including p gripping parts 31. np .

[0067] The second coordinate generation unit 155 generates the second coordinate information by, for example, synthesizing the robot joint angles shown in equation (5). 0 Q m is the second coordinate information V. In formula (5), j represents the order of joints from joint 32a to gripping portion 31. Therefore, in formula (5), m represents the number obtained by adding 1 (desired position) to the number of joints possessed by robot arm 30. The desired position is, for example, the position of the end of gripping portion 31. In formula (5), 0 represents the origin in the robot coordinate system. For example, 0 Q1 represents the displacement of the first joint (joint 32a) when viewed from the origin of the robot coordinate system. For example, j-1 Q j represents the displacement of the jth joint when viewed from the robot coordinate system of the j-1th joint. For example, m-1 Q m represents the displacement of the distal end of the gripping portion 31 when viewed from the robot coordinate system of the m-1th joint. Figure 1 The second coordinate information V of the robot arm 30 having six joints 32a to 32f and p gripping parts 31 is shown. n1 、V n2 ...V np When m=7 (the number of joints+1 (each gripping part)), j is an integer in the range of 0<j≤m.

[0068] [Formula 4]

[0069] 0 Q m = 0 Q1 1 Q2 2 Q3… j-i Q j … m-1 Q m ……(5)

[0070] In equation (6), the coordinate transformation matrix representing the displacement of each joint is j－1 Q j For example, it can be expressed as shown in the following formula (6). In formula (6), as in formula (3), for example, t is a translation vector with three rows and one column, and R is a rotation matrix with three rows and three columns. That is, the coordinate transformation matrix representing the displacement of each joint is j－1 Q jIt is also represented by a matrix of four rows and four columns. It should be noted that in formula (6), t represents the position of the joint and R represents the rotation of the joint.

[0071] [Formula 5]

[0072]

[0073] The second coordinate generating unit 155 generates, for example, n first coordinate information V'1, V'2, ..., V' for generating the captured image. n n second coordinate information V1, V2, ..., V of the gripping portion 31 of the robot arm 30 in the robot coordinate system V at the time point of each of the n images n It should be noted that, for example, when the robot control unit 153 moves the positions of the plurality of gripping units 31, the second coordinate generating unit 155 generates a plurality of second coordinate information V n For example, the second coordinate generation unit 155 generates n×p pieces of second coordinate information V based on n images containing p gripping parts 31. np .

[0074] The conversion information generation unit 156 estimates or updates the coordinate conversion matrix D that minimizes the error E using an approximation method in equation (7). In equation (7), X i The X component of the second coordinate information, X(D, X' i ) represents the X component of the multiplication result of the coordinate transformation matrix D and the first coordinate information. Similarly, Y i Represents the Y component of the second coordinate information, Y(D, Y' i ) represents the Y component of the multiplication result of the coordinate conversion matrix D and the first coordinate information. In addition, Z i Represents the Z component of the second coordinate information, Z(D, Z' i ) represents the Z component of the multiplication result of the coordinate conversion matrix D and the first coordinate information. That is, the conversion information generating unit 156 estimates the n first coordinate information V' by equation (7): n and n second coordinate information V n The coordinate transformation matrix D with the smallest error E between them is obtained. Approximation methods can use well-known numerical analysis methods such as Newton's method, gradient descent method, LM method, least squares method, etc. It should be noted that when estimating the coordinate transformation matrix D, it is preferred to set the initial value of the coordinate transformation matrix D to the smallest possible value.

[0075] [Formula 6]

[0076]

[0077] The conversion unit 157 converts the first coordinate information into the second coordinate information. The conversion unit 157 converts the first coordinate information V' into the second coordinate information V by Expression (2) using the coordinate conversion matrix D generated by the conversion information generation unit 156, for example.

[0078] (Process flow)

[0079] Next, the learning process and the correction process of the first embodiment will be described. Figure 6 and Figure 7 The learning process of the first embodiment will be described. Figure 6 is a flowchart showing one example of the learning process of the first embodiment. As shown in Figure 6 , first, the space generation unit 151 acquires a three-dimensional model of the gripping portion 31 of the robot arm 30 (step S101). Next, the space generation unit 151 sets conditions for arranging the gripping portion 31 of the robot arm 30 for which the three-dimensional model is acquired, i.e., the position and posture of the three-dimensional model, and the like (step S102). Then, the space generation unit 151 arranges the gripping portion 31 of the robot arm 30 for which the position and posture are determined on the virtual space (step S103).

[0080] Next, the learning unit 152 acquires a captured image, a position, and a posture of the gripping portion 31 of the robot arm 30, for example, by capturing the virtual space in which the three-dimensional model of the gripping portion 31 of the robot arm 30 is arranged (step S104). Then, the learning unit 152 saves the acquired image and the combination of the position and posture of the arranged gripping portion 31 as teaching data in the storage unit 14 (step S105). Further, the learning unit 152 repeatedly performs steps S102 to S105 a predetermined number of times (step S106: No). By repeatedly performing the processes of steps S102 to S105 a predetermined number of times, teaching data sufficient for repeating the learning process is generated.

[0081] The learning unit 152 performs the learning process using the generated teaching data when it is determined that the number of times of repeating steps S102 to S105 reaches the predetermined number (step S106: Yes) (step S107). Thus, the learning model 141 is generated or updated.

[0082] Next, the correction process in the correction system 1 will be described. Figure 7 is a flowchart showing one example of the correction process of the first embodiment. As shown in Figure 7 , first, the robot control unit 153 moves the gripping portion 31 of the robot arm 30 to a predetermined position (step S201). Next, the imaging device 20 images the work area including the gripping portion 31 and outputs it to the correction device 10 (step S202).

[0083] Next, the first coordinate generation section 154 generates first coordinate information V' representing the position 31' and the posture of the gripping portion 31 on the camera coordinate system V' using the learning model based on the image including the gripping portion 31 acquired from the imaging device 20 n (Step S203). Further, the second coordinate generation section 155 calculates second coordinate information V1 representing the position and the posture of the gripping portion 31 of the robot coordinate system V of the robot arm 30 based on the displacement of each joint angle of the robot arm 30 n (Step S204).

[0084] The conversion information generation section 156 determines whether the steps S201 to S204 are repeated for a prescribed number of times (step S210). In a case where it is determined that the prescribed number of times has not been reached (step S210: No), the processing is repeated back to the step S201.

[0085] The conversion information generation section 156, in a case where it is determined that the number of times that the steps S201 to S204 are repeated reaches the prescribed number of times (step S210: Yes), generates or updates the coordinate conversion matrix D using the first conversion information and the second conversion information (step S220).

[0086] As described above, the correction system 1 of the first embodiment is provided with a robot arm, an imaging device, and a correction device. The robot arm is provided with a gripping portion that grips an object to be gripped at a distal end thereof. The imaging device generates at least one captured image by capturing the robot arm from a prescribed direction. The correction device is provided with a first coordinate generation section and a conversion section. The first coordinate generation section generates first coordinate information representing the position of the gripping portion in a captured image coordinate system that is based on a captured image, based on a learning model that is generated by machine learning of a plurality of teaching data and the captured image captured from the prescribed direction, wherein the plurality of teaching data has an image that is captured by capturing a virtual space including a three-dimensional model related to the robot arm, and information representing the position of the gripping portion of the robot arm included in the virtual space. The conversion section coordinates-converts the first coordinate information into second coordinate information representing the position of the gripping portion in a robot coordinate system that is based on the robot arm, based on coordinate conversion information used in coordinate conversion processing of the first coordinate information into the second coordinate information. Thereby, it is possible to easily adjust the positional relationship between the robot arm and the camera. Further, the correction processing performed by the correction system 1 of the first embodiment can also be used in other correction systems having equivalent structures.

[0087] (Second Embodiment)

[0088] In the first embodiment, the structure of the correction system 1 is described in which the position of the robot arm 30 in the robot coordinate system V is calculated by synthesizing the displacements of the various joint angles of the robot arm 30. However, due to manufacturing errors in the robot arm 30, initialization deviations at power-on, etc., the position of the robot arm 30 in the robot coordinate system V during operation sometimes deviates from the position based on the design value of the robot arm 30. In the second embodiment, a structure is described in which coordinate conversion information is calculated while also taking into account errors in the rotation angle and translation position of the robot arm 30. It should be noted that, hereinafter, the same reference numerals are given to the same parts as those shown in the drawings described above, and repeated descriptions are omitted.

[0089] Figure 8 : is a block diagram showing an example of the configuration of the correction system according to the second embodiment. Figure 8 As shown, the calibration system 1a of the second embodiment includes a calibration device 10a instead of the calibration device 10 of the first embodiment. The processing unit 15a of the calibration device 10a includes a conversion information generating unit 156a instead of the conversion information generating unit 156 of the first embodiment. The processing unit 15a also includes an error adding unit 158a.

[0090] The error adding unit 158a generates the second coordinate information V generated by the second coordinate generating unit 155. n Give the error to generate the error second coordinate information V" n The error imparting unit 158a converts the second coordinate information V n The error imparting unit 158a decomposes the coordinate transformation matrix of each joint into the coordinate transformation matrix of each joint as shown in equation (5). j－1 Q j The rotation matrix j－1 R j Add an offset Δ to indicate the error j－1 R j Similarly, the error imparting unit 158a also applies the error to each coordinate transformation matrix. j－1 Q j The translation vector t is added to represent the error offset Δ j－1 t j .Added offset coordinate transformation matrices j－1 Q” j As shown in formula (8). It should be noted that, in the following, the offset Δ j－1 R j and Δ j－1 t j Simply record it as offset ΔR, Δt.

[0091] [Formula 7]

[0092]

[0093] The error assigning section 158a generates error second coordinate information Q" as shown in Expression (9) by synthesizing the coordinate conversion matrix as shown in Expression (8).

[0094] [Expression 8]

[0095] 0 Q" m 0 Q"1 1 Q"2 2 Q"3... j-1 Q" j ... m-1 Q" m ... (9)

[0096] The conversion information generating section 156a generates conversion information based on the first coordinate information and the error second coordinate information. The conversion information generating section 156a estimates or updates the coordinate conversion matrix D", the offset AR, and the offset At for which the error E is minimum, for example, using an approximation method or the like in Expression (10). In Expression (10), X" i represents the X component of the error second coordinate information, X(D", X' i ) represents the X component of the multiplication result of the coordinate conversion matrix D" and the first coordinate information. Similarly, Y" i represents the Y component of the error second coordinate information, Y(D", Y' i ) represents the Y component of the multiplication result of the coordinate conversion matrix D" and the first coordinate information. Further, Z" i represents the Z component of the error second coordinate information, Z(D", Z' i ) represents the Z component of the multiplication result of the coordinate conversion matrix D" and the first coordinate information. Note that, as with the case where the coordinate conversion matrix D is estimated separately, the initial values of the coordinate conversion matrix D", the offset AR, and At are preferably set to values as small as possible.

[0097] [Expression 9]

[0098]

[0099] Figure 9 is a flowchart showing one example of the correction processing of the second embodiment. Note that the processing in steps S201 to S210 is the same as the correction processing of the first embodiment, and thus detailed description is omitted. As shown in FIG. 18, the correction processing of the second embodiment includes steps S201 to S210 and a step S211. Figure 9 ​As shown, first, the error assigning section 158a decomposes the second coordinate information into the coordinate conversion matrix for each joint in a case where it is determined that the number of times of repeating steps S201 to S204 reaches the prescribed number (step S210: YES). Next, the error assigning section 158a appends the offsets AR and At to the rotation matrix and the translation vector of each of the decomposed coordinate conversion matrices, respectively (step S302). Then, the error assigning section 158a multiplies the coordinate conversion matrices to which the offsets AR and At are appended, and calculates the error second coordinate information V" (step S303).

[0100] Then, the conversion information generating section 156a generates the coordinate conversion matrix D" as the coordinate conversion information based on the first coordinate information V' and the error second coordinate information V" (step S320).

[0101] Thus, in the second embodiment, the second coordinate generating section generates the error second coordinate information including information related to errors of the rotation direction, the angle, and the translation position of the joint angle, and the coordinate conversion information generating section generates the coordinate conversion information based on the first coordinate information and the error second coordinate information. Thereby, when the coordinate conversion matrix D" is generated, not only the errors of the camera coordinate system V' and the robot coordinate system V but also the errors of the rotation direction, the angle, and the translation position caused by the robot arm 30 can be reflected.

[0102] (Modified Examples)

[0103] The above describes the embodiments of the present application, but the present application is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof. For example, it can be a structure in which the learning processing shown in Figure 6 FIG. 1, for example, is performed by other devices. That is, the space generating section 151 and the learning section 152 shown in Figure 3 FIG. 1 can be equipped in other devices than the correction device 10 shown in FIG. 1. In this case, the correction device 10 can also be a structure in which the learning model 141 is acquired from an external device and stored in the storage section 14. Figure 3

[0104] Further, it can be a structure in which the robot control processing is also performed by other devices. That is, the robot control section 153 shown in Figure 3 FIG. 1 can be equipped in other devices than the correction device 10. Also, it can be a structure in which the correction system 1 does not have the imaging device 20 and performs the correction processing using an image acquired from an external camera or the like.

[0105] ​Note that, in a case where the robot control section 153 moves the positions of the plurality (p) of grippers 31, for example, the first coordinate generation section 154 can also be configured to adopt only the gripper 31 corresponding to the value of the center among the grippers 31 detected with respect to one image as the first coordinate information V n In this case, the first coordinate generation section 154 generates n x 1 pieces of first coordinate information V n

[0106] Further, the various conditions at the time of arranging the three-dimensional models on the virtual space described above are not limited to the positions and postures of the three-dimensional models of the grippers 31 and the workpieces 41 and 42, and can be, for example, the setting of the angle of view, the number of pixels, the distance of the base line of the stereoscopic combination, and the like of the camera. Further, the various conditions can also be: the range of randomization of the position and rotation of the camera; the range of randomization of the brightness, angle, and position of the light source in the virtual space; the position, angle, number, and the like of the three-dimensional models of the grippers 31 and the workpieces 41 and 42 arranged on the virtual space. Also, the various conditions can also be the presence or absence of other obstacles such as a pallet on which the workpieces 41 and 42 are arranged, and the number, category, and the like of the obstacles. The category is information that uniquely identifies an arrangement on the virtual space. Note that the category in the various conditions is, for example, a combination of characters, numbers, and symbols, and the like. As long as it is information that does not duplicate other categories, the category can be any information. The arrangement is, for example, a three-dimensional model, a workpiece, or an obstacle.

[0107] Note that, Figure 1 A plurality of different kinds of workpieces 41 and 42 and the like are disclosed in the above-described embodiment, but the kind of workpiece can also be one kind. Further, the workpieces 41 and 42 and the like are arranged irregularly in position and posture, but can also be arranged regularly. Further, it can also be arranged so that, for example, a plurality of workpieces overlap in plan view as shown in FIG. 17. Figure 1

[0108] Further, the learning model 141 can also output angle information indicating the inclination of the gripper in addition to the coordinate indicating the position of the gripper. Note that, in the correction processing, the amount of rotation by which the robot control section 153 causes each joint 32a to 32f of the robot arm 30 to actuate can also be an amount that differs each time. Further, an example in which a convolutional neural network is used in the machine learning processing at the time of generating or optimizing the learning model 141 is described, but it is not limited thereto, and any algorithm can be used as long as it is a learning algorithm such as deep learning that can perform the above-described output.

[0109] ​​Note that the position of the gripping portion 31 of the robot arm 30 and the structure of the three-dimensional model used in the learning process and the correction process are described, but the position of a portion other than the gripping portion 31 of the robot arm 30 and the structure of the three-dimensional model can also be used. For example, as various conditions when the three-dimensional model is arranged on the virtual space, a condition in which the robot arm 30 becomes an obstacle when the gripping portion 31 is captured can be set.

[0110] Further, the order of the processes illustrated in the flowchart can be executed in sequence or simultaneously. For example, in the correction process illustrated in Figure 7 and Figure 9 the correction device 10 can generate the second coordinate information before the first coordinate information, or can generate the first coordinate information and the second coordinate information simultaneously in parallel. Further, the structure in which the robot arm 30 moves the gripping portion 31 before the photographing by the photographing device 20 is performed at the time of the correction process is described, but the embodiment is not limited thereto. For example, the structure in which the robot arm 30 moves the gripping portion 31 after the photographing by the photographing device 20 is performed and the first coordinate information and the second coordinate information are generated can also be used.

[0111] Further, the present application is not limited to the above-described embodiments. The application configured by appropriately combining each of the above-described components is also included in the present application. Further, a person skilled in the art can easily deduce further effects and modified examples. Therefore, the broader aspect of the present application is not limited to the above-described embodiments, and various modifications can be made.

[0112] Explanation of Reference Signs

[0113] 1: correction system; 10: correction device; 20: photographing device; 30: robot arm; 41, 42: workpiece.

Claims

1. A correction system including a robot arm, an imaging device, and a correction device, characterized by: the robot arm including a gripping portion that grips a gripping object at a distal end thereof; the imaging device being disposed at a position apart from the robot arm, and generating at least one captured image by capturing the robot arm from a predetermined direction; the correction device including: a first coordinate generation portion that generates first coordinate information indicating a position of the gripping portion in an imaging device coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the imaging device as a reference, based on a learning model generated by machine learning on a plurality of teaching data and the captured image obtained by capturing the robot arm from the predetermined direction, the plurality of teaching data including an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm, and information indicating a position of the gripping portion of the robot arm included in the virtual space; and a conversion portion that coordinate-converts the first coordinate information into second coordinate information indicating a position of the gripping portion in a robot coordinate system with the robot arm as a reference, based on coordinate conversion information used in coordinate conversion processing that coordinate-converts the first coordinate information into the second coordinate information. Further including: a robot control portion that drives the robot arm by a predetermined amount in accordance with the capturing from the predetermined direction; a second coordinate generation portion that generates the second coordinate information at a predetermined timing related to the capturing, based on joint angles of the robot arm; and a coordinate conversion information generation portion that generates the coordinate conversion information that coordinate-converts the first coordinate information into the second coordinate information, based on one or more of the first coordinate information and the same number of the second coordinate information as the first coordinate information generated based on the joint angles of the robot arm.

3. The correction system according to claim 2, characterized in that: the coordinate conversion information generation portion generates the coordinate conversion information in a manner that a difference between the first coordinate information and the second coordinate information satisfies a predetermined condition.

4. The correction system according to claim 2 or 3, characterized in that: the second coordinate generation portion generates error second coordinate information including information related to errors of a rotation direction and a translation position of the joint angles, the coordinate conversion information generation portion generates the coordinate conversion information based on the first coordinate information and the error second coordinate information.

5. An information processing system including a robot arm, an imaging device, and a correction device, characterized by: the robot arm including a gripping portion that grips a gripping object at a distal end thereof; the imaging device being disposed at a position apart from the robot arm, and generating at least one captured image by capturing the robot arm from a predetermined direction; the correction device including: ​ 2. The correction system of claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a first coordinate generation unit that generates first coordinate information indicating a position of the gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the camera as a reference, based on a learning model generated by machine learning on a plurality of teaching data and a captured image obtained by capturing the robot arm from a predetermined direction, the plurality of teaching data having an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm and information indicating a position of a gripping portion of a robot arm included in the virtual space; a second coordinate generation unit that generates second coordinate information at a predetermined timing related to the capturing, based on joint angles of the robot arm; and a coordinate conversion information generation unit that generates coordinate conversion information for converting the first coordinate information into the second coordinate information, based on one or more of the first coordinate information and the same number of second coordinate information as the first coordinate information generated based on the joint angles of the robot arm.

6. A robot control system including a robot arm, a camera, and a correction device, characterized by the robot arm including a gripping portion that grips an object at a distal end thereof, the camera being disposed at a position apart from the robot arm and generating at least one captured image by capturing the robot arm from a predetermined direction, the correction device including: a first coordinate generation unit that generates first coordinate information indicating a position of the gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the camera as a reference, based on a learning model generated by machine learning on a plurality of teaching data and a captured image obtained by capturing the robot arm from a predetermined direction, the plurality of teaching data having an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm and information indicating a position of a gripping portion of a robot arm included in the virtual space; a conversion unit that converts the first coordinate information into second coordinate information indicating a position of the gripping portion in a robot coordinate system with the robot arm as a reference, based on coordinate conversion information used in coordinate conversion processing for converting the first coordinate information into the second coordinate information and the first coordinate information; and a robot control unit that drives the robot arm by a predetermined amount in accordance with the capturing from the predetermined direction.

7. A correction method performed by a correction system including a robot arm including a gripping portion that grips an object at a distal end thereof, a camera disposed at a position apart from the robot arm and generating at least one captured image by capturing the robot arm from a predetermined direction, and a correction device, the correction method being characterized by Based on a learning model generated by machine learning on a plurality of teaching data and a captured image obtained by capturing the robot arm from a prescribed direction, first coordinate information indicating a position of the gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the camera as a reference is generated, wherein the plurality of teaching data having an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm and information indicating a position of a gripping portion of a robot arm included in the virtual space, The first coordinate information is coordinate-converted into second coordinate information based on coordinate conversion information used in a coordinate conversion process of coordinate-converting the first coordinate information into the second coordinate information and the first coordinate information, the second coordinate information indicating a position of the gripping portion in a robot coordinate system with the robot arm as a reference.

8. An information processing method executed by an information processing system that includes a robot arm having a gripping portion that grips an object to be gripped at a distal end thereof, an imaging device that is disposed at a position away from the robot arm and generates at least one captured image by capturing the robot arm from a predetermined direction, and a correction device, the information processing method characterized by, Based on a learning model generated by machine learning on a plurality of teaching data and a captured image obtained by capturing the robot arm from a prescribed direction, first coordinate information indicating a position of the gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the camera as a reference is generated, wherein the plurality of teaching data have an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm and information indicating a position of a gripping portion of a robot arm included in the virtual space, second coordinate information at a predetermined timing related to the capturing is generated based on joint angles of the robot arm, coordinate conversion information that coordinate-converts the first coordinate information into the second coordinate information is generated based on one or more of the first coordinate information and the same number of the second coordinate information generated based on the joint angles of the robot arm as the first coordinate information.

9. A robot control method executed by a robot control system that includes a robot arm having a gripping portion that grips an object to be gripped at a distal end thereof, an imaging device that is disposed at a position away from the robot arm and generates at least one captured image by capturing the robot arm from a predetermined direction, and a correction device, the robot control method characterized by, The first coordinate generation unit generates first coordinate information indicating a position of the gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with the camera as a reference, on the basis of a learning model generated by machine learning on a plurality of teaching data and a captured image obtained by capturing the robot arm from a prescribed direction, the plurality of teaching data have an image obtained by capturing a virtual space including a three-dimensional model related to the robot arm and information indicating a position of a gripping portion of a robot arm included in the virtual space; a conversion section that coordinate-converts the first coordinate information into second coordinate information based on coordinate conversion information used in a coordinate conversion process of coordinate-converting the first coordinate information into the second coordinate information and the first coordinate information, the second coordinate information indicating a position of the gripping portion in a robot coordinate system with the robot arm as a reference; and a robot control section that drives the robot arm by a predetermined amount in accordance with the capturing from the predetermined direction.

10. A correction device, characterized in that characterized by A first coordinate generation section generates first coordinate information indicating a position of a gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with reference to a camera, based on a learning model generated by machine learning on a plurality of teaching data and a captured image of the robot arm captured from a predetermined direction by a camera disposed at a position apart from the robot arm, the gripping portion gripping an object to be gripped, the robot arm having the gripping portion at a distal end thereof, the plurality of teaching data having an image captured of a virtual space including a three-dimensional model related to the robot arm and information indicating a position of the gripping portion of the robot arm included in the virtual space; and A second coordinate generation section coordinate-converts the first coordinate information into second coordinate information based on coordinate conversion information used in coordinate conversion processing of the first coordinate information into the second coordinate information, the second coordinate information indicating a position of the gripping portion in a robot coordinate system with reference to the robot arm.

11. An information processing apparatus comprising: Possessing: A first coordinate generation section generates first coordinate information indicating a position of a gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with reference to a camera, based on a learning model generated by machine learning on a plurality of teaching data and a captured image of the robot arm captured from a predetermined direction by a camera disposed at a position apart from the robot arm, the gripping portion gripping an object to be gripped, the robot arm having the gripping portion at a distal end thereof, the plurality of teaching data having an image captured of a virtual space including a three-dimensional model related to the robot arm and information indicating a position of the gripping portion of the robot arm included in the virtual space; A second coordinate generation section generates second coordinate information at a predetermined timing related to the capturing, based on joint angles of the robot arm; and A coordinate conversion information generation section generates coordinate conversion information for coordinate-converting the first coordinate information into the second coordinate information, based on one or more of the first coordinate information and the same number of the second coordinate information as the first coordinate information generated from the joint angles of the robot arm.

12. A robot control device characterized by comprising: Possessing: A first coordinate generation section generates first coordinate information indicating a position of a gripping portion in a camera coordinate system defined by an X' axis, a Y' axis, and a Z' axis with reference to a camera, based on a learning model generated by machine learning on a plurality of teaching data and a captured image of the robot arm captured from a predetermined direction by a camera disposed at a position apart from the robot arm, the gripping portion gripping an object to be gripped, the robot arm having the gripping portion at a distal end thereof, the plurality of teaching data having an image captured of a virtual space including a three-dimensional model related to the robot arm and information indicating a position of the gripping portion of the robot arm included in the virtual space; A second coordinate generation section generates second coordinate information at a predetermined timing related to the capturing, based on joint angles of the robot arm; and A coordinate conversion information generation section generates coordinate conversion information for coordinate-converting the first coordinate information into the second coordinate information, based on one or more of the first coordinate information and the same number of the second coordinate information as the first coordinate information generated from the joint angles of the robot arm. A conversion section converts the first coordinate information into second coordinate information based on coordinate conversion information used in a coordinate conversion process of converting the first coordinate information into the second coordinate information and the first coordinate information, the second coordinate information indicating a position of the gripping section in a robot coordinate system with the robot arm as a reference. A robot control section drives the robot arm by a prescribed amount based on the image taken from the prescribed direction.

Images