Control device, three-dimensional position measurement system, and program

Abstract

The control device is provided with: a combination generation unit that generates a plurality of combinations of three or more detection objects selected from among detection objects detected on the basis of an image obtained by capturing, by a visual sensor, three or more detection objects present on a workpiece, the mutual positional relationship of which is known; a selection unit that selects one or more combinations from among the plurality of combinations on the basis of indexes that are calculated for each of the plurality of generated combinations and that indicate positional shifts of the detection positions of the three or more detection targets from the ideal position; and a three-dimensional position determination unit that determines the three-dimensional position of the workpiece on the basis of the selected one or more combinations.

Description

Technical Field

[0001] The present disclosure relates to a control device, a three-dimensional position measurement system, and a program. Background Art

[0002] Various measurement systems for measuring the three-dimensional position of a three-dimensional object using a visual sensor have been proposed. For example, Patent Documents 1-2 describe a method in which three cameras are used to detect three detection objects whose relative positions are known in the three-dimensional object, and the three-dimensional position of the three-dimensional object is measured based on the detection positions of the three detection objects.

[0003] In relation to three-dimensional position measurement, Patent Document 3 describes a method for generating a three-dimensional model used in three-dimensional recognition processing using a stereo camera. Patent Document 4 describes an example of a method for three-dimensionally measuring the position and posture of an article conveyed by a conveyor.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 7-13613

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 62-54115

[0008] Patent Document 3: Japanese Patent Application Publication No. 2010-121999

[0009] Patent Document 4: Japanese Patent Application Publication No. 2019-128274 Summary of the invention

[0010] Problems to be solved by the invention

[0011] In the case of a structure in which three detection objects on a three-dimensional object are detected as described in Patent Documents 1 and 2, and the three-dimensional position of the three-dimensional object is obtained by the detection positions of the three detection objects, the measurement result may include errors in the positions of the detection objects themselves and measurement errors at the detection positions of the detection objects. Therefore, in the measurement of the three-dimensional object based on the detection positions of the three detection objects, sufficient accuracy may not be obtained sometimes. In addition, in the case in which the errors of some of the three detection objects are large, it is also conceivable that the overall error, that is, the three-dimensional position measurement result of the three-dimensional object, is enlarged due to the error.

[0012] A technology capable of reducing the influence of errors that may be included in the detection position of a detection target and thereby improving the accuracy of measuring the three-dimensional position of a three-dimensional object is desired.

[0013] Means for solving problems

[0014] One embodiment of the present disclosure is a control device comprising: a combination generating unit, which generates a plurality of combinations of three or more detection objects selected from detection objects detected based on images obtained by capturing three or more detection objects whose relative positional relationship is known and is present on a workpiece by a visual sensor; a selecting unit, which selects one or more combinations from the plurality of combinations based on indicators representing positional offsets of detection positions of the three or more detection objects relative to an ideal position, respectively calculated for the plurality of generated combinations; and a three-dimensional position determining unit, which determines the three-dimensional position of the workpiece based on the one or more selected combinations.

[0015] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of typical embodiments of the present invention as shown in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a diagram showing the equipment configuration of a robot system including a robot control device according to an embodiment.

[0017] Figure 2 FIG. 1 is a diagram showing a vehicle body as an example of a workpiece and an inspection target.

[0018] Figure 3 This is a diagram showing the visual coordinate system and the sensor coordinate system assigned to each point located at the zero deviation position on the workpiece.

[0019] Figure 4 It is a diagram showing the sensor coordinate system and the projection of the points onto the image plane.

[0020] Figure 5 This is a functional block diagram of a robot control device and an image processing device.

[0021] Figure 6 This is a flowchart showing the basic operation of the three-dimensional position measurement process. DETAILED DESCRIPTION

[0022] Next, the embodiments of the present disclosure are described with reference to the accompanying drawings. In the referenced drawings, the same components or functional parts are marked with the same reference symbols. For easy understanding, the scales of these drawings are appropriately changed. In addition, the method shown in the drawings is an example for implementing the present invention, and the present invention is not limited to the method shown in the drawings.

[0023] Figure 1 1 is a diagram showing the device configuration of a robot system 100 including a robot control device 50 according to an embodiment. Figure 1As shown, the robot system 100 includes a robot 10, a visual sensor 70 mounted on the front end of the hand of the robot 10, a robot control device 50 for controlling the robot 10, a teaching operation panel 40, and an image processing device 20. The teaching operation panel 40 and the image processing device 20 are connected to the robot control device 50. The visual sensor 70 is connected to the image processing device 20. The robot system 100 of this embodiment is configured as a three-dimensional position measurement system, which can measure the three-dimensional position of the workpiece W with high accuracy by detecting three or more detection targets on the three-dimensional object placed on the table 1 (a carriage on a conveying device, a stage, etc.), that is, the workpiece W.

[0024] The robot 10 is a vertical multi-joint robot. In addition, other types of robots such as a horizontal multi-joint robot, a parallel link robot, a dual-arm robot, etc. may be used as the robot 10 depending on the work object. The robot 10 can perform the desired work through the end effector installed on the wrist. The end effector is an external device that can be replaced according to the purpose, such as a hand, a welding gun, a tool, etc. Figure 1 , an example using the hand 33 as an end effector is shown.

[0025] The robot control device 50 controls the operation of the robot 10 according to the operation program or the instruction from the teaching operation panel 40. The robot control device 50 may also have a hardware structure as a general computer, and the computer has a processor 51 ( Figure 5 ), memory (ROM, RAM, non-volatile memory, etc.), storage device, operating unit, input and output interface, network interface, etc.

[0026] The image processing device 20 has a function of controlling the visual sensor 70 and a function of performing image processing including object detection processing, etc. The image processing device 20 may have a hardware structure as a general computer having a processor, a memory (ROM, RAM, nonvolatile memory, etc.), a storage device, an operation unit, a display unit, an input / output interface, a network interface, etc.

[0027] In addition, Figure 1 , a configuration example is described in which an image processing device that is responsible for controlling the visual sensor 70 and processing images is configured as an independent device in the robot system 100, but the function of the image processing device 20 can also be integrated into the robot control device 50.

[0028] The teaching operation panel 40 is used as an operation terminal for teaching and performing various settings of the robot 10. As the teaching operation panel 40, a teaching device composed of a tablet terminal or the like may be used. The teaching operation panel 40 may also have a hardware structure as a general computer, which has a processor, a memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, and a display unit 41 ( Figure 5 ), input and output interfaces, network interfaces, etc.

[0029] The workpiece W to be measured in three dimensions is, for example, Figure 2 The vehicle body shown. On the workpiece W, three or more detection objects (e.g., circular holes M) are provided at positions whose relative positions are known. These detection objects are arranged, for example, on the bottom surface of the vehicle body. The robot system 100 detects the positions of the three or more detection objects by means of the visual sensor 70, thereby calculating the three-dimensional position of the entire workpiece W. The robot system 100 can obtain the three-dimensional position of the workpiece W and appropriately perform various operations on the workpiece W.

[0030] exist Figure 1 , a configuration example in which the visual sensor 70 is mounted on the fingertips of the robot 10 is shown. In this configuration, the robot 10 moves the visual sensor 70, positions the visual sensor 70 at various imaging positions for imaging the detection object (circular hole M), and images and detects the detection object. The robot 10 may also be taught in advance the imaging positions of the detection objects that can image the workpiece W located at the reference position.

[0031] Instead of mounting the visual sensor on the robot 10, the structure may be configured so that the detection object is photographed and detected by one or more visual sensors fixedly arranged in the working space. In this case, multiple visual sensors that respectively photograph multiple detection objects on the workpiece may be configured. Alternatively, a configuration in which one visual sensor photographs more than two detection objects may be configured. In the latter case, the number of visual sensors configured can be less than the total number of detection objects.

[0032] Regarding the imaging position (posture) of the visual sensor 70 on the detection object, the image planes following any imaging position (posture) of the visual sensor cannot be mutually constrained. In addition, it is preferable that the normal vectors to any image planes form a relatively large angle with each other.

[0033] The robot system 100 (robot control device 50) detects the positions of three or more detection objects on the workpiece W, and obtains the three-dimensional position of the workpiece W based on the detected positions. The following describes a method of detecting the positions of three detection objects on the workpiece as a basic detection method, and then describes a method of expanding the method to four or more detection objects. On this basis, the determination of the three-dimensional position of a three-dimensional object based on the detected positions of three or more detection objects is described.

[0034] A method for detecting three detection objects on the workpiece W and obtaining the three-dimensional position of the workpiece will be described. The "position detection function" for detecting the positions of the three detection objects on the workpiece W can also be implemented as the image processing unit (detection unit) 121 ( Figure 5 ) function. Figure 3 As shown, the workpiece W can be considered as a rigid body with three known points (i.e., detection objects). When the workpiece W is at the zero deviation position, i.e., the ideal nominal position, the local coordinate system with the origin located on or near the workpiece W is considered, i.e., the visual coordinate system (hereinafter, also recorded as VCS). At the points corresponding to each detection object (hereinafter, also recorded as punctuation points) at the position assigned with zero deviation, three orthogonal vectors are established with these points as starting points, the magnitudes of these vectors are set to unit length, and their directions are parallel to the directions of the three vectors of the visual coordinate system VCS. The small coordinate system formed by the three unit vectors at each point is called sensor coordinate system 1, 2, and 3 (also recorded as SCS1, SCS2, and SCS3, respectively). The transformation of these three sensor coordinate systems remains unchanged.

[0035] The visual coordinate system VCS is fixed relative to the imaging position (posture) of the visual sensor 70. The coordinate system fixed to the workpiece W is called the workpiece coordinate system (also recorded as BCS). When the workpiece W is in its zero deviation position, each mark accurately corresponds to each origin of the three sensor coordinate systems.

[0036] When the workpiece W moves from its zero offset position, the rigid body motion to which the workpiece W is subjected is completely determined by the transformation [T] relating VCS to BCS. This transformation is defined with respect to VCS and completely determines the position and orientation of BCS, and therefore the position of the workpiece W.

[0037] Given the zero-deviation position coordinates of a point in the VCS and the position coordinates of the point when it is displaced, the zero-deviation position coordinates and the position coordinates after displacement are directly related through the transformation [T]. The purpose of the three-dimensional position determination function described below is to be able to determine the transformation [T] by detecting the point within the field of view of the visual sensor 70 at each shooting position.

[0038] When the workpiece W is in a position with some deviation, the mark on the image plane moves to a position away from the origin of the SCS coordinate system. Figure 4 The SCS1 coordinate system and the projection of the point P1 onto the image plane in this case are shown in . In general, by combining three or more projections onto the image plane with the calibration data, the six degrees of freedom of the deviation of the three-dimensional object relative to the nominal position can be determined. By assuming that point P1 is located on the XY plane of the SCS1 coordinate system, the position of point P1 can be independently determined from the images captured at each camera position. Vectors A, B, and P are defined as follows. u is the horizontal axis on the image plane, and v is the vertical axis on the image plane. The capped u and v are unit vectors in the horizontal and vertical axis directions of the image plane, respectively.

[0039] [Formula 1]

[0040]

[0041] Vector A and vector B are projections of the unit vector in the X direction and the unit vector in the Y direction in the SCS1 coordinate system onto the image plane. The X coordinate and Y coordinate of point P1 (ie, x1 and y1) are given by equations (1) to (4) shown below.

[0042] [Formula 2]

[0043] P=x1A+y1B...(1)

[0044]

[0045] Reference Figure 4 , in a more general case, it is represented by the following equations (5) to (9). In this case, it is assumed that z can take any value.

[0046] [Formula 3]

[0047]

[0048] P′=x1A+y1B...(9)

[0049] Based on these equations (5) to (9), equations (10) to (11) are obtained, and the solution is obtained by expressing x1 and y1 with z1.

[0050] [Formula 4]

[0051]

[0052] u PO -z1u CO =x1u AO +y1u BO

[0053] vPO -z1v CO =x1v AO +y1v BO ...(11)

[0054]

[0055] The above x1 and y1 are rewritten as follows.

[0056] [Formula 5]

[0057]

[0058] Here, α1, β1, γ1, and δ1 are constants given by the following formula.

[0059] [Formula 6]

[0060]

[0061] Equation (12) indicates that both x1 and y1 are linear functions of z1. The same equations are derived for the other two image planes. The complete set of equations is given by equations (13) to (15). The constants appearing in equations (13) to (15) can be obtained by calibration using a calibration tool. As a calibration tool, for example, a cube having edges and scales corresponding to mutually orthogonal coordinate axes of the SCS coordinate system is positioned in such a way that the three edges are parallel to the mutually orthogonal coordinate axes of the SCS coordinate system. Then, the cube is photographed at the position and posture of the photographing mark (SCS coordinate system) by the visual sensor 70, and information related to the actual size of the cube can be used to obtain information (calibration data) about what vector the unit vectors of the X, Y, and Z axes of the SCS coordinate system correspond to on the image. Such calibration data is pre-stored in the storage unit 122 ( Figure 5 )wait.

[0062] [Formula 7]

[0063] x1=α1z1+β1

[0064] y1=γ1z1+δ1...(13)

[0065] x2=α2z2+β2

[0066] y2=γ2z2+δ2...(14)

[0067] x3=α3z3+β3

[0068] y3=γ3z3+δ3...(15)

[0069] Equations (13) to (15) are six linear equations with nine unknowns. To solve these equations, the workpiece is considered as a rigid body as an additional constraint. That is, here, the distance between the points on the workpiece is fixed. The origin of each SCS coordinate system is represented by (X O1 , Y O1 , Z O1 )、(X O2 , Y O2 , Z O2 )、(X O3 , Y O3 , Z O3 ), the coordinates of each mark after displacement are expressed as P1 (X1, Y1, Z1), P2 (X2, Y2, Z2), P3 (X3, Y3, Z3). The distances between the origins of the three SCS coordinate systems are expressed as follows. In addition, the distances between each mark after displacement are given as formula (16).

[0070] [Formula 8]

[0071]

[0072] d 12 =[(x O1 -x O2 ) 2 +(y O1 -y O2 ) 2 +(z O1 -z O2 ) 2 ] 1 / 2

[0073] d 12 =[x O12 2 +y O12 2 +z O12 2 ] 1 / 2

[0074] d 23 =[x O23 2 +y O23 2 +z O23 2 ] 1 / 2

[0075] d 31 =[x O31 2 +y O31 2 +z O31 2] 1 / 2

[0076] [Formula 9]

[0077]

[0078] By substituting equations (13) to (15) into equation (16), the following first set of equations (equation (17)) is obtained. In addition, rewriting these equations yields the second set of equations (equation (18)). In these equations, k, l, and m are constants.

[0079] [Formula 10]

[0080]

[0081] [Formula 11]

[0082]

[0083] The second set of equations (Equation (18)) is solved, for example, by using Newton's iterative method. After these values ​​are obtained, they are substituted into equations (13) to (15) to obtain x1, x2, x3 and y1, y2, y3. The (x1, y1, z1), (x2, y2, z2), (x3, y3, z3) thus obtained are the positions of each point on the SCS coordinate system after displacement. They can be transformed into values ​​on the VSC. Thus, a transformation [T] associating VCS with BCS can be obtained. That is, the three-dimensional position of the workpiece W after displacement is obtained.

[0084] In addition, in the above method, it is assumed that the three points are projected orthographically on the image plane. Considering that the actual projection is close to the perspective projection, a process of correcting the error of the mapping relationship can be performed. In order to compensate for the error, a mapping relationship given by the following equation is established between the actual coordinate axis and its respective projection axis. The mapping relationship for each axis can be obtained by measuring more than 3 points on each coordinate axis during calibration and using an interpolation method to obtain the required relationship.

[0085] [Formula 12]

[0086] x′1=f 11 (x1),y′1=f 12 (y1),z′1=f 13 (z1)

[0087] x′2=f 21 (x2),y′2=f 22 (y2),z′2=f 23 (z2)

[0088] x′3=f31 (x3),y′3=f 32 (y3),z′3=f 33 (z3)

[0089] λ1=x1 / x′1 μ1=y1 / y′1 η1=z1 / z′1

[0090] λ2=x2 / x′2 μ2=y2 / y′2 η2=z2 / z′2

[0091] λ3=x3 / x′3 μ3=y3 / y′3 η3=z3 / z′3

[0092] The calculation result of the new scale factor is shown below.

[0093] [Formula 13]

[0094]

[0095] The above calculation method uses the least squares method to calculate the position deviation of the three points. At the end of the least squares calculation, the projection of the coordinate axis is multiplied by a new scale factor to compensate for nonlinearity. Using the new scale factors given by the three sets of equations, the above constants α, β, γ, and δ are recalculated for each image plane. After that, the calculation based on the least squares method is performed again.

[0096] Consider extending the above calculation method to the case where more than four punctuation marks are used. Here, the case where four punctuation marks are used is described. As described above, assuming that each punctuation mark is located on the XY plane of the SCS coordinate system, the equations described in the above formulas (1) to (4) are established for the four punctuation marks, thereby obtaining the position of each punctuation mark.

[0097] Regarding a more general case, similarly to the case where equations (13) to (15) are obtained for three punctuation points, the x and y coordinates of four punctuation points can be expressed as linear functions of z as shown in the following equations (19) to (22).

[0098] [Formula 14]

[0099] x1=α1z1+β1

[0100] y1=γ1z1+δ1...(19)

[0101] x2=α2z2+β2

[0102] y2=γ2z2+δ2...(20)

[0103] x3=α3z3+β3

[0104] y3=γ3z3+δ3...(21)

[0105] x4=α4z4+β4

[0106] y4=γ4z4+δ4...(22)

[0107] Next, based on the fact that the workpiece W is a rigid body and the distances between the origins of the four SCS coordinate systems are equal to the distances between the four measuring points (marking points), the following equations are obtained regarding the distances between the origins of the four SCS coordinate systems and the distances between the four marking points. In addition, here, as the distances between the origins of the four SCS coordinate systems (the distances between the four marking points), equations related to d12, d23, d34, and d41 are established to obtain the solutions of equations (19) to (22), but it is also possible to further establish equations related to d13 and d24 as the distances between the origins of the four SCS coordinate systems (the distances between the four marking points), and obtain the solutions of equations (19) to (22) by taking these equations into consideration.

[0108] [Formula 15]

[0109] d 12 =[x O12 2 +y O12 2 +z O12 2 ] 1 / 2

[0110] d 23 =[x O23 2 +yO232+z232]1 / 2

[0111] d 34 =[x O34 2 +y O34 2 +z O34 2 ] 1 / 2

[0112] d 41 =[x O41 2 +y O41 2 +z O41 2 ] 1 / 2

[0113] [Formula 16]

[0114]

[0115] By substituting equations (19) to (22) into equation (23), equations (24) and (25) are obtained by expanding equations (17) and (18) for four punctuation marks as described below.

[0116] [Formula 17]

[0117]

[0118] [Formula 18]

[0119]

[0120] By solving this equation by the iterative method in the same way as in the above method, the positions of x1, x2, x3, x4 and y1, y2, y3, y4, i.e., the shifted positions of the four reference points, can be obtained. Then, the three-dimensional position of the workpiece W is obtained by synthesizing the detected positions of the four reference points. That is, based on these detected positions, the transformation [T] that relates VCS to BCS is obtained.

[0121] It is understandable that the same approach can be used to expand the measurement when measuring a further increased number of detection objects (punctuation points).

[0122] In addition, various methods can be used as a method for determining the three-dimensional position of the workpiece W based on the detection positions of more than three detection objects (marks). As an example, the following various methods can be applied. In addition, in the method exemplified below, when there is a condition related to the configuration of the detection object (mark), the condition is followed.

[0123] (1) A method of finding the parameters of the above transformation [T] (parameters representing translation and rotation) by solving simultaneous equations.

[0124] (2) As described in Patent Document 4 (Japanese Patent Publication No. 2019-128274), the position and posture of a workpiece are determined by matching a polygon of a known shape (a polygon connecting punctuation points at zero deviation positions) to the line of sight of a camera relative to the detection position of each punctuation point.

[0125] (3) A method of determining a plane of a coordinate system (such as an XY plane) on a workpiece based on the positions of three or more marks on the workpiece to grasp the coordinate system. In this case, for example, the coordinate system is grasped by setting the first mark as the origin, the second mark as the position in the X-axis direction, and the third mark (and the fourth and subsequent marks) as representing the position on the XY plane.

[0126] The calculation function of obtaining the three-dimensional position of the workpiece W based on the detection positions of three or more detection objects (marks) can be implemented as a function in the selection unit 153 or the three-dimensional position determination unit 154 in the robot control device 50.

[0127] Figure 5 is a functional block diagram of the robot control device 50 and the image processing device 20. Figure 5 As shown, the robot control device 50 includes an action control unit 151, a combination generation unit 152, a selection unit 153, and a three-dimensional position determination unit 154. In addition, these functional blocks may be implemented by executing a program by the processor 51 of the robot control device 50. In addition, the robot control device 50 includes a storage unit 155.

[0128] The storage unit 155 is constituted by, for example, a nonvolatile memory, a hard disk device, etc. The storage unit 155 stores an operation program for controlling the robot 10 , a program (vision program) for image processing such as workpiece detection based on an image captured by the vision sensor 70 , and various setting information.

[0129] The motion control unit 151 controls the motion of the robot according to the motion program of the robot. The robot control device 50 includes a servo control unit (not shown) that performs servo control of the servo motor for each axis according to the command for each axis generated by the motion control unit 151. The motion control unit 151 has the function of moving the visual sensor 70 and positioning it at the shooting position for shooting each detection object.

[0130] The combination generating unit 152 provides a function of generating a plurality of combinations of three or more detection targets selected from the detection targets detected on the workpiece W.

[0131] The selection unit 153 provides a function of selecting one or more combinations from a plurality of combinations based on the “offset” calculated from each of the plurality of generated combinations.

[0132] The three-dimensional position determination unit 154 provides a function of determining three-dimensional position information of the workpiece W based on one or more combinations of detection targets selected by the selection unit 153. Details of the functions of the combination generation unit 152, the selection unit 153, and the three-dimensional position determination unit 154 will be described later.

[0133] The image processing device 20 includes an image processing unit 121 and a storage unit 122. The storage unit 122 is a storage device composed of, for example, a nonvolatile memory. The storage unit 122 stores various data required for image processing, such as shape data and correction data of the detection object. The image processing unit 121 performs various image processing such as detection processing of the workpiece. That is, the image processing unit 121 has a function as a detection unit that detects the detection object on the image captured by the visual sensor 70 within the imaging range including the detection object.

[0134] The three-dimensional measurement function of the workpiece W by the robot controller 50 will be described. Figure 6 This is a flowchart showing the basic operation of the three-dimensional position measurement process executed under the control of the robot control device 50 (processor 51).

[0135] First, the image processing unit (detection unit) 121 detects the detection object based on the image obtained by the visual sensor 70 taking the detection object (step S1). Here, the robot 10 positions the visual sensor 70 at the shooting position for shooting each detection object, and shoots an image including the detection object. The image processing unit (detection unit) 121 obtains the respective positions (x, y) of more than three detection objects through the above-mentioned position detection function.

[0136] Next, the combination generation unit 152 generates a plurality of combinations in which three or more detection objects are selected from the detected detection objects (step S2). For example, the combination generation unit 152 may also generate all possible combinations based on the three or more detection objects detected. In this case, for example, if the number of the detected detection objects is five, the number of possible combinations is the total number of combinations using all five detection objects, the number of combinations using four of the five detection objects, and the number of combinations using three of the five detection objects.

[0137] Alternatively, the combination generating unit 152 may generate a combination of detection targets according to the following rule.

[0138] (Rule 1) Select the object to be excluded from the three or more detected objects, among which at least three are retained.

[0139] (Rule 2) You can also specify the maximum number of detection objects to be excluded.

[0140] (Rule 3) The minimum number of remaining detection objects can also be specified.

[0141] When selecting the detection object to be excluded, by changing the detection object to be excluded, a combination of multiple detection objects can be generated. For example, when 8 detection objects are detected, if the maximum number to be excluded is specified as 2, (1+8+8×7÷2=37) combinations of detection results are generated.

[0142] The combination generation unit 152 may also be configured to accept input (input from an external device or user input) of "selection of detection objects to be excluded", "maximum number of detection objects to be excluded", or "minimum number of detection objects remaining". The user interface for accepting user input may also be presented on the display unit 41 of the teaching operation panel 40. User input may also be performed via the operation unit of the teaching operation panel 40. The combination generation unit 152 may also generate a combination using values ​​pre-set in the robot control device 50 for "selection of detection objects to be excluded", "maximum number of detection objects to be excluded", or "minimum number of detection objects remaining".

[0143] In this way, by incorporating more detection objects into the calculation of the overall three-dimensional position (the three-dimensional position of the workpiece W), the impact of errors that may be contained in each detection object can be reduced as a whole, thereby improving the measurement accuracy of the three-dimensional position.

[0144] Next, the selection unit 153 calculates the overall three-dimensional position (the three-dimensional position of the workpiece W) and an index indicating the positional deviation of the detection positions of the three or more detection objects included in the combination from the ideal position (hereinafter, this index is referred to as "positional deviation") for each combination of the generated detection objects. Then, the selection unit 153 selects one or more combinations based on the "positional deviation" (step S3).

[0145] As an example, the selection unit 153 calculates the "position offset" as follows. Assume that the overall three-dimensional position is obtained as position A for a certain combination. Using the designed position Pi of the i-th detection object on the workpiece W, the ideal position of the detection object when the three-dimensional position of the workpiece W is position A is obtained as A·Pi. The number of detection objects in the combination is set to n. The selection unit 153 can also calculate the position offset D based on the difference Ki between A·Pi and the position P'i of the i-th detection object (mark) after displacement obtained by the above formula (25). For example, the selection unit 153 can calculate the position offset D as the average value ΣKi / n of Ki. In this case, for a certain combination, the position offset D becomes an indicator of the extent to which the detection position of the detection object included in the combination is offset from the ideal position. Alternatively, the selection unit 153 can also calculate the position offset D based on the distance Di between the line of sight Li toward the actual detection position of the i-th detection object and A·Pi. For example, the selection unit 153 can also calculate the position offset D as the average ΣDi / n of Di. In this case, the position deviation D also serves as an indicator of the extent to which the detection position of the detection target included in the combination is deviated from the ideal position with respect to a certain combination.

[0146] The selection unit 153 can select one or more combinations based on the positional deviation D calculated for each of the generated combinations. In this case, the selection unit 153 can select a combination using a selection criterion that (r1) the smaller the positional deviation D, the better the accuracy.

[0147] Therefore, for example, the selection unit 153 may select a predetermined number of combinations whose values ​​of the positional deviation D are small, or may select one combination whose value of the positional deviation D is the smallest.

[0148] In this way, by adopting a structure that selects a combination for calculating the overall three-dimensional position (three-dimensional position of the workpiece W) based on the position deviation D, it is possible to exclude combinations with a high possibility of having a large error, thereby improving the measurement accuracy of the three-dimensional position.

[0149] Next, the three-dimensional position determination unit 154 determines the final three-dimensional position of the workpiece W based on the one or more combinations selected by the selection unit 153 (step S4). When the combination selected by the selection unit 153 is one, the three-dimensional position determination unit 154 may determine the position A of the workpiece W obtained by the one combination as the final three-dimensional position of the workpiece W.

[0150] When there are multiple combinations selected by the selection unit 153, the three-dimensional position determination unit 154 may determine the final three-dimensional position of the workpiece W based on the statistics related to the three-dimensional positions of the workpiece W obtained for each of the multiple combinations. For example, the three-dimensional position determination unit 154 may determine the final three-dimensional position of the workpiece W as the average value or the median value of the three-dimensional positions of the workpiece W obtained for each of the selected multiple combinations.

[0151] As described above, according to the three-dimensional position measurement process of this embodiment, the influence of errors can be reduced and the measurement accuracy of the three-dimensional position of the three-dimensional object can be improved.

[0152] When selecting a combination in step S3 of the above three-dimensional position measurement process, the selection unit 153 may further consider the number of detection objects in each generated combination. In this case, the selection unit 153 may also use the following selection criteria for selection:

[0153] (r1) The smaller the position offset D, the better the accuracy; and

[0154] (r2) The more detection objects there are in the combination, the better the accuracy.

[0155] In addition, the selection criterion (r2) in this case is based on the fact that the greater the number of detection objects, the more errors that may be included in each object can be offset, thereby improving the overall position measurement accuracy.

[0156] As an example, assume that there are a plurality of selection candidates with good (relatively small) combinations of positional deviations D. In this case, the selection unit 153 may select a larger combination of one or more detection targets from the plurality of selection candidates.

[0157] When generating a combination in step S2 of the above-mentioned three-dimensional position measurement process, the combination generating unit 152 may also output a combination in which a specific combination is selected from the combinations that can be generated based on the detected detection objects as the generated combination. For example, consider a situation in which the number of detection objects detected in step S1 is large. In this case, the number of combinations that can be generated becomes very large. In such a situation, the combination generating unit 152 may also output a combination randomly selected from all combinations that can be generated. Thus, a combination can be selected from a plurality of combination candidates without bias for use.

[0158] In step S3 of the above three-dimensional position measurement process, consider the situation where there are multiple combinations selected based on the position offset D, or the position offset D and the number of detection objects. In this case, the number of selected combinations can be reduced by further repeating the processing of steps S2 to S3 for the selected combination more than once. In this case, the following processing is performed more than once:

[0159] (1) based on the detection objects included in the one or more combinations selected by the selection unit 153, the combination generation unit 152 generates a plurality of combinations (second plurality of combinations) in which three or more detection objects are selected again.

[0160] (2) Based on the indices (position shifts) calculated for the second plurality of combinations, the selection unit 153 again selects one or more combinations from the second plurality of combinations.

[0161] For example, it is assumed that the number of detected detection objects is 20, and in the generation of the first combination in the combination generation unit 152, when the combination is generated according to the rule "the minimum number of remaining detection objects is set to 10", the number of combinations selected by the selection unit 153 becomes a considerable number. In this case, the combination generation unit 152 can generate the second plurality of combinations by applying a rule such as "the minimum number of remaining detection objects is set to 15" to the detection objects included in the combination selected by the selection unit 153. However, in this case, the second plurality of combinations is generated in the form of selecting combinations that meet the rule of "the minimum number of remaining detection objects is set to 15" by taking the combinations pre-selected by the selection unit 153 as the parent set. The selection unit 153 can also select a combination from the second plurality of combinations based on the above-mentioned selection criterion (r1) or the above-mentioned selection criterion (r1) and (r2).

[0162] Furthermore, the narrowing down of the selections by repeating the generation of the combination by the combination generation unit 152 and the selection by the selection unit 153 may be performed as follows.

[0163] The combination generation unit 152 regenerates a plurality of combinations including three or more detection objects by deleting one or more detection positions that satisfy the criterion that the index indicating the positional offset calculated for a certain detection position (for example, the above-mentioned Ki or Di) is larger than the index indicating the positional offset calculated for other detection positions from the one or more combinations selected by the selection unit 153, and performs this operation one or more times until the index indicating the positional offset calculated for the detection objects in each regenerated combination satisfies a predetermined condition. In this case, the predetermined condition may be an average value of the index indicating the positional offset of the detection objects in each regenerated combination, or a value of the index being less than a predetermined value.

[0164] Specifically, the operation may be as follows.

[0165] The operation (b1) of the combination generating unit 152 regenerates a plurality of combinations including more than three detection objects by deleting one or more detection positions satisfying the criterion that "the difference Ki calculated for a certain detection position is greater than the difference Ki calculated for other detection positions" from the one or more combinations selected by the selecting unit 153, and is performed one or more times in such a manner that (b2) ΣKi / n or Ki for the generated combinations becomes less than a predetermined value. In the above (b1), for example, a predetermined number of detection objects with large difference Ki may be deleted from the detection objects included in a certain combination.

[0166] Alternatively, the narrowing down of the selections by repeating the generation of the combination by the combination generation unit 152 and the selection by the selection unit 153 may be performed as follows.

[0167] The operation (c1) of the combination generating unit 152 regenerates a plurality of combinations including more than three detection objects by deleting one or more detection positions satisfying the criterion that "the distance Di calculated for a certain detection position is greater than the distance Di calculated for other detection positions" from one or more combinations selected by the selecting unit 153, and is performed one or more times in such a manner that ΣDi / n or Di for the generated combinations becomes less than a predetermined value. In the above (c1), for example, a predetermined number of detection objects with a large distance Di may be deleted from the detection objects included in a certain combination.

[0168] By repeating the selection structure in this way, it is possible to quickly narrow down appropriate selection candidates, especially when the number of detection objects is large.

[0169] As described above, according to the present embodiment, the influence of errors that may be included in the detection position of the detection target can be reduced, thereby improving the accuracy of the three-dimensional position measurement of the three-dimensional object.

[0170] Figure 3 The functional configuration in the functional block diagram is an example, and various modifications are possible regarding the functional allocation within the robot system 100. For example, a configuration example in which a part of the functions in the robot control device 50 is arranged on the teaching operation panel 40 side is also possible.

[0171] The teaching pendant 40 and the robot controller 50 may be defined as a robot controller as a whole.

[0172] The configuration of the robot control device in the above-described embodiment (including the case where the function of the image processing device is integrated) can be applied to control devices for various industrial machines.

[0173] Figure 5The functional blocks of the robot control device and the image processing device shown above may be realized by the processors of these devices executing various software stored in the storage device, or may be realized by a configuration mainly composed of hardware such as ASIC (Application Specific Integrated Circuit).

[0174] The program for executing various processing such as the three-dimensional position measurement processing in the above-mentioned embodiment can be recorded in various computer-readable recording media (for example, semiconductor memories such as ROM, EEPROM, flash memory, magnetic recording media, CD-ROM, DVD-ROM and other optical disks).

[0175] The present disclosure has been described in detail, but the present disclosure is not limited to the above-mentioned embodiments. These embodiments can be variously added, replaced, changed, partially deleted, etc. without departing from the scope of the main purpose of the present disclosure, or without departing from the scope of the main purpose of the present disclosure derived from the contents recorded in the scope of the patent protection requested and its equivalents. In addition, these embodiments can also be implemented in combination. For example, in the above-mentioned embodiments, the order of each action and the order of each processing are shown as an example and are not limited to this. In addition, the same is true for the case where numerical values ​​or mathematical formulas are used in the description of the above-mentioned embodiments.

[0176] The following supplementary notes are further described with respect to the above-mentioned embodiment and modified examples.

[0177] (Note 1)

[0178] A control device (50) comprises: a combination generating unit (152) which generates a plurality of combinations of three or more detection objects selected from detection objects detected based on an image obtained by photographing three or more detection objects existing on a workpiece and having a known relative positional relationship with each other by a visual sensor (70); a selecting unit (153) which selects one or more combinations from the plurality of combinations based on an index representing a positional offset of the detection positions of the three or more detection objects relative to an ideal position calculated for each of the plurality of generated combinations; and a three-dimensional position determining unit (154) which determines the three-dimensional position of the workpiece based on the one or more selected combinations.

[0179] (Note 2)

[0180] A control device (50) according to Supplementary Note 1, wherein the combination generating unit (152) generates all possible combinations based on the detected detection objects.

[0181] (Note 3)

[0182] The control device (50) according to Supplementary Note 1, wherein the combination generating unit (152) generates a plurality of the combinations by excluding or selecting a predetermined number of detection objects from the detected detection objects.

[0183] (Note 4)

[0184] The control device (50) according to Supplementary Note 1, wherein the combination generating unit (152) generates a plurality of the combinations by randomly selecting from the combinations that can be generated based on the detected detection objects.

[0185] (Note 5)

[0186] The control device (50) according to any one of Supplementary Notes 1 to 4, wherein for each of the generated plurality of combinations, the selection unit (153) performs the following processing:

[0187] (1) when the three-dimensional position of the workpiece obtained based on one combination is set as position A and the designed position of the i-th detection object on the workpiece is set as Pi, the ideal position of the i-th detection object on the workpiece is obtained as A·Pi; and

[0188] (2) For each detection object in the one combination, a difference Ki between the detection position P'i of the i-th detection object in the one combination and A·Pi is obtained, and the index is obtained based on the obtained difference Ki.

[0189] (Note 6)

[0190] A control device (50) according to Supplementary Note 5, wherein when the number of detection objects in the one combination is set to n, the selection unit (153) obtains an average value of the differences Ki, that is, ΣKi / n, as the index.

[0191] (Note 7)

[0192] The control device (50) according to any one of Supplementary Notes 1 to 4, wherein for each of the generated plurality of combinations, the selection unit (153) performs the following processing:

[0193] (1) when the three-dimensional position of the workpiece obtained based on one combination is set as position A and the designed position of the i-th detection object on the workpiece is set as Pi, the ideal position of the i-th detection object on the workpiece is obtained as A·Pi; and

[0194] (2) For each detection object in the combination, the distance Di between the line of sight Li from the visual sensor to the detection position of the i-th detection object in the combination and A·Pi is calculated, and the index is calculated based on the calculated distance Di.

[0195] (Note 8)

[0196] According to the control device (50) described in Supplementary Note 7, when the number of detection objects in the one combination is set to n, the selection unit (153) obtains the average value of the distance Di, that is, ΣDi / n, as the indicator.

[0197] (Note 9)

[0198] The control device (50) according to any one of Supplementary Notes 1 to 8, wherein the selection unit (153) selects the one or more combinations using a selection criterion that the smaller the size of the index, the better the accuracy.

[0199] (Note 10)

[0200] A control device (50) according to any one of Notes 1 to 8, wherein the selection unit (153) selects one or more combinations from the multiple combinations based on the indicators calculated for the multiple combinations respectively and the number of detection objects of each combination in the multiple combinations.

[0201] (Note 11)

[0202] The control device (50) according to Supplementary Note 10, wherein, for each of the plurality of combinations, the selection unit (153) selects the one or more combinations using the following selection criteria:

[0203] (1) The smaller the size of the indicator, the better the accuracy; and

[0204] (2) The greater the number of detection objects in the combination, the better the accuracy.

[0205] (Note 12)

[0206] A control device (50) according to any one of Notes 1 to 11, wherein the three-dimensional position determination unit (154) determines the three-dimensional position of the workpiece based on statistics of the three-dimensional positions of the workpiece obtained from the one or more selected combinations.

[0207] (Note 13)

[0208] According to the control device (50) described in Appendix 12, wherein the three-dimensional position determination unit (154) determines the three-dimensional position of the workpiece as an average value or a median value of the three-dimensional positions of the workpiece obtained by the one or more selected combinations.

[0209] (Note 14)

[0210] The control device (50) according to any one of Supplementary Notes 1 to 13, wherein the control device performs the following actions one or more times:

[0211] The combination generating unit (152) regenerates a plurality of combinations in which more than three detection objects are selected based on the detection objects included in the one or more combinations selected by the selecting unit (153), and the selecting unit (153) reselects one or more combinations from the regenerated plurality of combinations based on the indicators respectively calculated for the regenerated plurality of combinations.

[0212] (Note 15)

[0213] A control device (50) according to any one of Notes 1 to 4, wherein the combination generating unit (152) regenerates a plurality of combinations including more than three detection objects by deleting one or more detection positions that satisfy a benchmark that an indicator indicating the position offset calculated for a certain detection position is greater than an indicator indicating the position offset calculated for other detection positions from the one or more combinations selected by the selecting unit (153), and the process is executed one or more times until the indicator indicating the position offset calculated for the detection objects in each regenerated combination satisfies a predetermined condition.

[0214] (Note 16)

[0215] A control device (50) according to Supplementary Note 15, wherein the predetermined condition is an average value of an indicator representing the positional deviation of the detection object in each regenerated combination, or a value of the indicator is less than a predetermined value.

[0216] (Note 17)

[0217] A three-dimensional position measurement system (100) comprises: a visual sensor (70); a detection unit (121) which detects three or more detection objects whose relative positions are known and which exist on a workpiece based on an image captured by the visual sensor; a combination generation unit (152) which generates a plurality of combinations of three or more detection objects selected from the detected detection objects; a selection unit (153) which selects one or more combinations from the plurality of combinations based on indicators representing positional deviations of the detection positions of the three or more detection objects relative to an ideal position, calculated for the plurality of generated combinations; and a three-dimensional position determination unit (154) which determines the three-dimensional position of the workpiece based on the one or more selected combinations.

[0218] (Note 18)

[0219] The three-dimensional position measurement system (100) described in Note 17 also comprises: a robot (10) equipped with the visual sensor (70); and a motion control unit (151) which controls the robot (10) to position the visual sensor (70) at a shooting position for respectively shooting the three or more detection objects.

[0220] (Note 19)

[0221] A program for causing a computer processor to execute the following steps: detecting three or more detection objects whose relative positions are known and existing on a workpiece based on an image captured by a visual sensor (70); generating a plurality of combinations of three or more detection objects selected from the detected detection objects; selecting one or more combinations from the plurality of combinations based on indices representing positional deviations of the detection positions of the three or more detection objects relative to an ideal position, calculated for each of the plurality of generated combinations; and determining the three-dimensional position of the workpiece based on the one or more selected combinations.

[0222] Explanation of symbols

[0223] 1 unit

[0224] 10. Robot

[0225] 20 Image processing device

[0226] 33 Hands

[0227] 40 Teaching Operation Panel

[0228] 41 Display unit

[0229] 50Robot control device

[0230] 51 processors

[0231] 70 Vision Sensors

[0232] 100 Robotic Systems

[0233] 121 Image Processing Department

[0234] 122 Storage Department

[0235] 151 Motion Control Unit

[0236] 152 Combination Generation Department

[0237] 153 Selection Department

[0238] 154 Three-dimensional position determination unit

[0239] 155 storage unit.

Claims

1. A control device, characterized in that: have: a combination generating unit for generating a plurality of combinations of three or more detection objects selected from detection objects detected based on an image obtained by capturing three or more detection objects existing on a workpiece and having a known mutual positional relationship with each other by a visual sensor; a selection unit that selects one or more combinations from the plurality of combinations based on indices indicating positional deviations of the detection positions of the three or more detection objects relative to ideal positions, calculated for the plurality of combinations generated; and A three-dimensional position determination unit determines the three-dimensional position of the workpiece based on the one or more selected combinations.

2. The control device according to claim 1, characterized in that: The combination generating unit generates all possible combinations according to the detected detection objects.

3. The control device according to claim 1, characterized in that: The combination generating unit generates the plurality of combinations by excluding or selecting a predetermined number of detection objects from the detected detection objects.

4. The control device according to claim 1, characterized in that: The combination generating unit generates the plurality of combinations by randomly selecting from the combinations that can be generated based on the detected detection objects.

5. The control device according to any one of claims 1 to 4, characterized in that: For each of the generated multiple combinations, the selection unit performs the following processing: (1) When the three-dimensional position of the workpiece obtained based on one combination is set as position A and the designed position of the i-th detection object on the workpiece is set as Pi, the ideal position of the i-th detection object on the workpiece is obtained as A·Pi; as well as (2) For each detection object in the one combination, a difference Ki between the detection position P'i of the i-th detection object in the one combination and A·Pi is obtained, and the index is obtained based on the obtained difference Ki.

6. The control device according to claim 5, characterized in that: When the number of detection objects in the one combination is n, the selection unit obtains ΣKi / n, which is an average value of the differences Ki, as the index.

7. The control device according to any one of claims 1 to 4, characterized in that: For each of the generated multiple combinations, the selection unit performs the following processing: (1) When the three-dimensional position of the workpiece obtained based on one combination is set as position A and the designed position of the i-th detection object on the workpiece is set as Pi, the ideal position of the i-th detection object on the workpiece is obtained as A·Pi; as well as (2) For each detection object in the combination, the distance Di between the line of sight Li from the visual sensor to the detection position of the i-th detection object in the combination and A·Pi is calculated, and the index is calculated based on the calculated distance Di.

8. The control device according to claim 7, characterized in that: When the number of detection objects in the one combination is n, the selection unit obtains ΣDi / n, which is an average value of the distances Di, as the index.

9. The control device according to any one of claims 1 to 8, characterized in that: The selection unit selects the one or more combinations using a selection criterion that the smaller the index is, the higher the accuracy is.

10. The control device according to any one of claims 1 to 8, characterized in that: The selection unit selects one or more combinations from the plurality of combinations based on the indices respectively calculated for the plurality of combinations and the number of detection targets of each of the plurality of combinations.

11. The control device according to claim 10, characterized in that: For each of the plurality of combinations, the selection unit selects the one or more combinations using the following selection criteria: (1) The smaller the size of the indicator, the better the accuracy; and (2) The greater the number of detection objects in the combination, the better the accuracy.

12. The control device according to any one of claims 1 to 11, characterized in that: The three-dimensional position determination unit determines the three-dimensional position of the workpiece based on statistics of the three-dimensional positions of the workpiece obtained from the one or more selected combinations.

13. The control device according to claim 12, characterized in that: The three-dimensional position determination unit determines an average value or a median value of the three-dimensional positions of the workpiece obtained from the one or more selected combinations as the three-dimensional position of the workpiece.

14. The control device according to any one of claims 1 to 13, characterized in that: The control device performs the following actions more than once: the combination generating unit regenerates a plurality of combinations in which three or more detection objects are selected, based on the detection objects included in the one or more combinations selected by the selecting unit; and The selection unit selects one or more combinations from the regenerated plurality of combinations based on the indexes respectively calculated for the regenerated plurality of combinations.

15. The control device according to any one of claims 1 to 4, characterized in that: The combination generating unit regenerates a plurality of combinations including more than three detection objects by deleting one or more detection positions satisfying a benchmark that an index representing the position offset calculated for a certain detection position is greater than an index representing the position offset calculated for other detection positions from the one or more combinations selected by the selecting unit, and executes this operation more than once until an index representing the position offset calculated for the detection objects in each regenerated combination satisfies a predetermined condition.

16. The control device according to claim 15, characterized in that: The predetermined condition is an average value of an index indicating the positional deviation of the detection object in each regenerated combination, or a value of the index being equal to or less than a predetermined value.

17. A three-dimensional position measurement system, characterized in that: have: Vision sensors; a detection unit that detects three or more detection objects existing on the workpiece and having a known mutual positional relationship based on the image captured by the visual sensor; a combination generating unit for generating a plurality of combinations of three or more detection objects selected from the detected detection objects; a selection unit that selects one or more combinations from the plurality of combinations based on indices indicating positional deviations of the detection positions of the three or more detection objects relative to ideal positions, calculated for the plurality of combinations generated; and A three-dimensional position determination unit determines the three-dimensional position of the workpiece based on the one or more selected combinations.

18. The three-dimensional position measurement system according to claim 17, characterized in that: Also available: A robot equipped with the visual sensor; and The motion control unit controls the robot to position the visual sensor at a photographing position for photographing the three or more detection objects respectively.

19. A program for causing a processor of a computer to perform the following steps: Detecting three or more detection objects existing on the workpiece and having a known mutual positional relationship based on an image captured by a visual sensor; generating a plurality of combinations of three or more detection objects selected from the detected detection objects; selecting one or more combinations from the plurality of combinations based on indices indicating positional deviations of the detection positions of the three or more detection objects relative to ideal positions, calculated for the plurality of combinations generated respectively; and The three-dimensional position of the workpiece is determined based on the one or more selected combinations.

Images