Position and Orientation Measurement System

Abstract

Provided is a system that is capable of accurately measuring the three-dimensional position and posture of an object under measurement with simple teaching and in a short measurement time. A position and posture measurement system according to the present invention is provided with: a vision sensor that has already been calibrated; a position and posture moving unit that is capable of moving the three-dimensional position and posture of an object under measurement or the vision sensor; a calibration-data storage unit; an image processing unit that processes a captured image of the object under measurement; a position and posture measurement unit that measures the three-dimensional position and posture of the object under measurement by using the result of the image processing; an image-capturing-position correction unit that executes correction processing at least once, the correction processing being processing in which the vision sensor or the object under measurement is moved by the position and posture moving unit by using the measured three-dimensional position and posture and in which the image processing and the measurement of the three-dimensional position and posture are then sequentially carried out; and an image-capturing-position correction termination unit that executes the correction processing again upon determining that the correction processing is not to be terminated, while adopting the three-dimensional position and posture calculated either the last or in the middle as a three-dimensional position and posture to be used for robot control upon determining that the correction processing is to be terminated.

Description

[Technical field]

[0001] The present disclosure relates to a position and orientation measurement system. [Background technology]

[0002] Conventionally, workpieces are processed by a robot mounted on a transport device such as a cart. After the robot is moved to the vicinity of the workpiece by the transport device, it performs various tasks such as loading / unloading the workpiece, changing tools, and processing the workpiece. However, if the stopping position or stopping posture of the robot changes, the position and posture of the robot relative to the working space also change, so the robot may not be able to work properly if it only performs the same operation every time. Therefore, it is necessary to measure the deviation of the position and posture of the robot relative to the working space and correct the operation of the robot.

[0003] For example, a technology has been proposed in which a visual sensor attached to the robot's hand or the like is used to detect the three-dimensional positions of multiple reference points installed in the workspace, and the deviation amount from the robot's reference position and reference posture relative to the workspace is calculated as a correction amount, and the robot's movement is corrected (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 11-156764 [Patent Document 2] JP 2019-093481 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the techniques of Patent Documents 1 and 2 have problems in that the 3D position and orientation of the measurement object determined from the captured image varies widely, or the number of imaging position teaching points increases because multiple measurement objects arranged in multiple locations are imaged, which increases the time required for teaching and measurement. Therefore, the conventional techniques cannot be used for applications that require precise operations such as workpiece mounting with short teaching and measurement times.

[0006] An object of the present disclosure is to provide a position and orientation measurement system that can accurately measure the three-dimensional position and orientation of a measurement target object with easier teaching and shorter measurement time than conventional systems. [Means for solving the problem]

[0007] One aspect of the present disclosure is a position and orientation measurement system that measures a three-dimensional position and orientation of a measurement object used to control a robot, the system including a calibrated visual sensor, a position and orientation moving unit capable of moving the three-dimensional position and orientation of the measurement object or the visual sensor, a calibration data storage unit that pre-stores calibration data obtained when calibrating the visual sensor, an image processing unit that processes an image obtained by capturing an image of the measurement object with the visual sensor, a position and orientation measurement unit that measures the three-dimensional position and orientation of the measurement object using a result of image processing by the image processing unit, and a calibration data storage unit that stores calibration data obtained by calibrating the visual sensor ... an imaging position correction unit that performs at least one correction process in which the visual sensor or the object to be measured is moved by the position and orientation moving unit using the three-dimensional position and orientation of the object to be measured, and then sequentially performs image processing by the image processing unit and measurement by the position and orientation measurement unit, using the three-dimensional position and orientation of the object to be measured; and an imaging position correction ending unit that determines whether or not to end the correction process by the imaging position correction unit, and if it is determined that the correction process should not be ended, executes the correction process again, and if it is determined that the correction process should be ended, adopts any one of the three-dimensional position and orientation of the object to be measured last or during the process by the position and orientation measurement unit as the three-dimensional position and orientation of the object to be measured to be used for control of the robot. Effect of the Invention

[0008] According to one aspect of the present disclosure, it is possible to provide a position and orientation measurement system that can accurately measure the three-dimensional position and orientation of a measurement target object with easier teaching and shorter measurement time than conventional systems. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a configuration of a robot system according to a first embodiment. [Diagram 2] 1 is a block diagram showing the configuration of a position and orientation measurement system according to a first embodiment. [Diagram 3] 5 is a flowchart showing a processing procedure of the position and orientation measurement system according to the first embodiment. [Figure 4] 11A and 11B are diagrams showing changes in an input image when an imaging position is moved. [Diagram 5] 11A and 11B are diagrams showing the position of the measurement object before the imaging position is moved and the position of the measurement object when the reference position is set. [Figure 6] 13A and 13B are diagrams showing the position of the measurement object after the imaging position has been moved and the position of the measurement object when the reference position is set. [Figure 7] 10 is a flowchart showing a processing procedure of a position and orientation measurement system according to a modified example of the first embodiment. [Figure 8] FIG. 11 is a block diagram showing the configuration of a position and orientation measurement system according to a second embodiment. [Figure 9] 11A and 11B are diagrams showing changes in an input image when the imaging position is moved and the luminance is adjusted. [Figure 10] FIG. 13 is a block diagram showing the configuration of a position and orientation measurement system according to a third embodiment. [Figure 11] 13 is a diagram showing the display screen of the interface device when it is determined that the difference in luminance of the measurement object before and after movement of the imaging position is greater than a second threshold value. FIG. [Figure 12] FIG. 13 is a block diagram showing the configuration of a position and orientation measurement system according to a fourth embodiment. [Figure 13]13 is a diagram showing the display screen of the interface device when it is determined that the undetectable range of the input image after the imaging position is moved is larger than the undetectable threshold value. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of each embodiment, the same reference numerals will be used to designate configurations common to the first embodiment, and descriptions thereof will be omitted as appropriate.

[0011] [First embodiment] The position and orientation measurement system according to the first embodiment of the present disclosure is a system that measures the three-dimensional position and orientation (X, Y, Z, W, P, R) of a measurement object used to control the operation of a robot, using an image of the measurement object captured by a calibrated visual sensor and calibration data. More specifically, the position and orientation measurement system according to the first embodiment is a system that can reduce measurement variability compared to when no correction is performed, by performing correction to bring the relative position between the current imaging position and the measurement object closer to the relative position between the imaging position and the measurement object when the robot's reference position is set, and can measure the three-dimensional position and orientation of the measurement object more accurately than in the past.

[0012] Here, the reference position of the robot means the position that serves as the reference for making corrections. Correction is performed by calculating the difference between the position of the object to be measured when this reference position is set (reference position) and the position of the object to be measured when correction is performed, and applying this difference to the movement to be corrected. The difference is expressed by a homogeneous transformation matrix, and the corrected position is obtained by multiplying it by the three-dimensional position (vector). A similar correction can also be performed by storing the three-dimensional position of the object to be measured when the reference position is set, teaching the robot's movement using that position as the reference coordinate system, and rewriting the coordinate system with the position of the object at the time of measurement.

[0013] Fig. 1 is a diagram showing the configuration of a robot system 100 according to the first embodiment. As shown in Fig. 1, the robot system 100 is configured to include a position and orientation measurement system 1 according to the first embodiment. The robot system 100 includes a robot 2, a visual sensor 4, a visual sensor control device 20, a robot control device 10, and a measurement target 7. The robot system 100 recognizes the three-dimensional position and orientation of a workpiece based on, for example, a captured image of the workpiece captured by the visual sensor 4, and executes a predetermined task such as handling or machining of the workpiece.

[0014] A hand tool 6 is attached to the tip of the robot arm 3 of the robot 2 via a flange 5. The robot 2 executes a predetermined task such as handling or machining of a workpiece under the control of a robot control device 10. In addition, a visual sensor 4 is attached to the tip of the robot arm 3 of the robot 2 via a flange 5. This allows the visual sensor 4 to move in a three-dimensional position and orientation.

[0015] The visual sensor 4 is controlled by a visual sensor control device 20, and captures an image of a measurement target 7 such as a workpiece, a calibration jig, or a marker. A calibrated visual sensor is used as the visual sensor 4. As the visual sensor 4, a general two-dimensional camera or a three-dimensional camera such as a stereo camera may be used.

[0016] The visual sensor control device 20 controls the visual sensor 4 and processes the images captured by the visual sensor 4. In addition, the visual sensor control device 20 detects the three-dimensional position and orientation of the measurement object 7 from the images captured by the visual sensor 4.

[0017] The robot control device 10 executes an operation program for the robot 2 and controls the operation of the robot 2. In so doing, the robot control device 10 corrects the operation of the robot 2 based on the three-dimensional position and orientation of the measurement target 7 detected by the visual sensor control device 20 so that the robot 2 performs a predetermined task.

[0018] Furthermore, the robot control device 10 controls the position and attitude of the robot 2 so as to control the position and attitude of the visual sensor 4 when capturing an image by the visual sensor 4. In this manner, the robot control device 10 controls the relative position between the measurement object 7 and the visual sensor 4 by fixing the position and attitude of the measurement object 7 and controlling the position and attitude of the visual sensor 4.

[0019] Fig. 2 is a block diagram showing the configuration of the position and orientation measurement system 1 according to the first embodiment. As shown in Fig. 2, the position and orientation measurement system 1 according to the first embodiment includes a visual sensor control device 20 and a robot control device 10. The visual sensor control device 20 includes an image processing unit 21, a calibration data storage unit 22, and an image acquisition unit 23. The robot control device 10 includes a position and orientation measurement unit 11, an imaging position correction unit 12, an imaging position correction completion unit 13, and a position and orientation movement unit 14.

[0020] The image acquisition unit 23 acquires, as an input image, a captured image captured by the visual sensor 4. The image acquisition unit 23 transmits the acquired input image to the image processing unit 21.

[0021] The image processing unit 21 performs image processing on the input image transmitted from the image acquisition unit 23. More specifically, the image processing unit 21 detects the measurement object 7 from the input image when a detection score based on the degree of match with a pre-stored model pattern is equal to or greater than a predetermined threshold value, for example, by using the model pattern stored in advance. The image processing unit 21 also obtains the three-dimensional position and orientation of the measurement object 7 with respect to the visual sensor 4 from the input image and calibration data stored in a calibration data storage unit 22 described later.

[0022] The principle of determining the three-dimensional position and orientation of the measurement object 7 with respect to the visual sensor 4 from the position information of the measurement object 7 such as a calibration jig in an input image captured using a calibrated visual sensor 4 is well known, for example, from "Computer Vision, Technical Review and Future Outlook" by Matsuyama Takashi. That is, the geometric transformation characteristics inside the visual sensor 4 and the geometric relationship between the three-dimensional space in which the object exists and the two-dimensional image plane are already determined, and further, multiple pairs of information of the positions on the two-dimensional image of features such as multiple dots displayed on the calibration jig and the positions in the three-dimensional space are obtained, so that the relative positions of the visual sensor 4 and the measurement object 7 can be uniquely determined. Therefore, it is possible to determine the three-dimensional position and orientation of the measurement object 7 with respect to the visual sensor 4 from the position information of the measurement object 7 in an input image captured using a calibrated visual sensor 4.

[0023] The image processing unit 21 in the visual sensor control device 20 and the robot control device 10 are configured with an arithmetic processor such as a DSP (Digital Signal Processor) or an FPGA (Field-Programmable Gate Array). The image processing unit 21 in the visual sensor control device 20 and the various functions of the robot control device 10 are realized by executing, for example, predetermined software (programs, applications). The image processing unit 21 in the visual sensor control device 20 and the various functions of the robot control device 10 may be realized by a combination of hardware and software, or may be realized only by hardware (electronic circuits).

[0024] The calibration data storage unit 22 pre-stores calibration data obtained when performing calibration of the visual sensor 4. The calibration data storage unit 22 in the visual sensor control device 20 is configured with a rewritable memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory).

[0025] The calibration data of the visual sensor 4 stores internal parameters and external parameters of the visual sensor 4. Examples of the internal parameters of the visual sensor 4 include parameters such as lens distortion and focal length. Examples of the external parameters of the visual sensor 4 include the three-dimensional position and orientation of the visual sensor 4 based on the reference position of the robot 2 when the reference position of the robot 2 is set, and the three-dimensional position and orientation of the visual sensor 4 based on the position of the flange 5.

[0026] The position and orientation measurement unit 11 measures the three-dimensional position and orientation of the measurement object 7 using the image processing result by the image processing unit 21. Specifically, the position and orientation measurement unit 11 determines the three-dimensional position and orientation of the measurement object 7 based on the reference position of the robot 2 at the time of imaging, from the three-dimensional position and orientation of the visual sensor 4 based on the reference position of the robot 2 at the time of imaging (i.e., the current position and orientation of the visual sensor 4) and the three-dimensional position and orientation of the measurement object 7 based on the visual sensor 4 determined by the image processing unit 21.

[0027] The three-dimensional position and orientation of the visual sensor 4 based on the reference position of the robot 2 at the time of imaging is obtained as follows. Since the visual sensor 4 in this embodiment is a hand camera or the like, when the robot 2 moves, the position of the visual sensor 4 also changes accordingly. Therefore, first, the three-dimensional position and orientation of the visual sensor 4 based on the flange 5, which does not change depending on the movement of the robot 2 after the visual sensor 4 is attached, is obtained from the calibration data stored in the calibration data storage unit 22. In addition, the three-dimensional position and orientation of the flange 5 based on the reference position of the robot 2 at the time of imaging, which can always be obtained from the robot control device 10 that generates the operation command for the robot 2, is obtained. Then, the three-dimensional position and orientation of the visual sensor 4 based on the reference position of the robot 2 at the time of imaging is calculated and obtained based on the obtained three-dimensional position and orientation of the visual sensor 4 based on the flange 5 and the obtained three-dimensional position and orientation of the flange 5 based on the reference position of the robot 2 at the time of imaging.

[0028] The imaging position correction unit 12 uses the three-dimensional position and orientation of the measurement target 7 measured by the position and orientation measurement unit 11 to move the visual sensor 4 by a position and orientation moving unit 14 described below, and then sequentially causes the image processing unit 21 to perform image processing and the position and orientation measurement unit 11 to perform measurement. The imaging position correction unit 12 executes the correction processing consisting of the movement of the visual sensor 4, the captured image processing, and the three-dimensional position and orientation measurement at least once.

[0029] Furthermore, it is preferable that the imaging position correction unit 12 executes the above correction process by moving the robot 2 to a robot position where the visual sensor 4 approaches the position of the visual sensor 4 as seen from the object to be measured when the reference position is set.

[0030] This correction process by the imaging position correction unit 12 brings the current relative position between the imaging position and the measurement object 7 closer to the relative position between the imaging position and the measurement object 7 when the reference position of the robot 2 is set. This suppresses variation in the measurement of the three-dimensional position and orientation of the measurement object 7 by the position and orientation measurement unit 11. That is, in the past, there was a problem that the variation in the three-dimensional position and orientation of the measurement object found from the captured image was large, and if the measurement object and the position to be corrected were to change significantly away from each other, the correction error would increase, but this embodiment can solve this problem. Therefore, this embodiment can be applied to applications that require precise operations, such as mounting a workpiece.

[0031] The imaging position correction ending unit 13 determines whether or not to end the correction process by the imaging position correction unit 12. The imaging position correction ending unit 13 of the present embodiment determines whether or not to end the correction process depending on whether or not the correction process by the imaging position correction unit 12 has been executed a predetermined number of times. The predetermined number of times is set to, for example, three times.

[0032] Specifically, when the imaging position correction end unit 13 determines that the correction process by the imaging position correction unit 12 has not been performed a predetermined number of times and that the correction process should not be ended, the imaging position correction end unit 13 causes the imaging position correction unit 12 to perform the correction process again.

[0033] In addition, when the imaging position correction end unit 13 determines that the correction process by the imaging position correction unit 12 has been executed a predetermined number of times and that the correction process is to be ended, it adopts any one of the three-dimensional position and orientation of the measurement object 7 measured last or during by the position and orientation measurement unit 11 as the three-dimensional position and orientation of the measurement object 7 to be used for controlling the robot 2.

[0034] The three-dimensional position and orientation of the measurement object 7 based on the reference position of the robot 2, which is finally measured by the position and orientation measurement unit 11 in the correction process, is an accurate three-dimensional position and orientation in which measurement variation is suppressed. Therefore, by controlling the operation of the robot 2 using the three-dimensional position and orientation of the measurement object 7 based on the reference position of the robot 2 adopted by the imaging position correction completion unit 13, accurate operation control of the robot 2 becomes possible.

[0035] Furthermore, when the amount of movement of the visual sensor 4 in the correction process is very small (for example, smaller than a predetermined movement amount threshold), the accuracy of the movement of the robot 2 itself has a greater effect on the correction error. Therefore, in this case, the correction error may be suppressed by adopting, for example, the 3D position and orientation measured just before the last one of the 3D positions and orientations of the measurement target object 7 measured during the correction process.

[0036] For example, the three-dimensional position and orientation of the measurement object 7 based on the reference position of the robot 2 adopted by the imaging position correction end unit 13 can be used to control the operation of the robot 2 with respect to the object to which the measurement object 7 is attached, and also to set a coordinate system used for controlling the robot 2. In this case, the movement amount of the coordinate system obtained by moving and rotating the coordinate system itself so that the set coordinate system overlaps with the base coordinate system at the reference position can be used as a shift amount, i.e., a correction amount, to correct the operation of the robot 2. In addition, it can also be used as a coordinate system used for setting the teaching position of the robot 2. In the example shown in FIG. 1, the measurement object 7 is placed on a table T, and in this case, the coordinate system used for controlling the robot 2 is set to the measurement object 7 on the table T.

[0037] The position and orientation moving unit 14 moves the three-dimensional position and orientation of the visual sensor 4. Specifically, the position and orientation moving unit 14 moves the three-dimensional position and orientation of the visual sensor 4 by controlling the operation of the robot 2. The imaging position correction unit 12 described above executes the correction process by moving the visual sensor 4 using the position and orientation moving unit 14.

[0038] The processing of the position and orientation measurement system 1 according to the first embodiment having the above configuration will be described in detail with reference to Fig. 3. Fig. 3 is a flowchart showing the procedure of the processing of the position and orientation measurement system 1 according to the first embodiment.

[0039] First, in step S11, an input image is acquired. Specifically, the measurement target 7 is imaged by the visual sensor 4, and the input image is acquired by the image acquisition unit 23. After that, the process proceeds to step S12.

[0040] Next, in step S12, the three-dimensional position and orientation of the measurement object 7 is measured. Specifically, the position and orientation measurement unit 11 measures the three-dimensional position and orientation of the measurement object 7 using the image processing result of the input image by the image processing unit 21. More specifically, the position and orientation measurement unit 11 determines the three-dimensional position and orientation of the measurement object 7 based on the reference position of the robot 2 at the time of imaging, from the three-dimensional position and orientation of the visual sensor 4 based on the reference position of the robot 2 at the time of imaging, and the three-dimensional position and orientation of the measurement object 7 based on the visual sensor 4 determined by the image processing unit 21. Thereafter, the process proceeds to step S13.

[0041] Next, in step S13, the visual sensor 4 is moved to a position when the reference position of the robot 2 was set. Specifically, the imaging position correction unit 12 operates the robot 2 using the three-dimensional position and orientation of the measurement target object 7 measured by the position and orientation measurement unit 11, and moves the visual sensor 4 to the position of the visual sensor 4 when the reference position of the robot 2 was set. Then, the process proceeds to step S14.

[0042] Next, in step S14, an input image is acquired. Specifically, the measurement target 7 is imaged again by the visual sensor 4 moved in step S13, thereby acquiring the input image in the image acquisition unit 23. Then, the process proceeds to step S15.

[0043] Next, in step S15, the three-dimensional position and orientation of the measurement object 7 is measured. Specifically, the position and orientation measurement unit 11 measures the three-dimensional position and orientation of the measurement object 7 using the image processing result by the image processing unit 21 of the input image of the measurement object 7 captured in step S14 using the visual sensor 4 moved in step S13. Note that the procedure for measuring the three-dimensional position and orientation of the measurement object 7 in the position and orientation measurement unit 11 is the same as in step S12. Then, the process proceeds to step S16.

[0044] Next, in step S16, it is determined whether the image capture position has been corrected a predetermined number of times, for example, three times. If the determination is NO, the process returns to step S13, and the image capture position correction unit 12 again executes the correction process, which includes moving the visual sensor 4, processing the captured image, and measuring the three-dimensional position and orientation. If the determination is YES, the process proceeds to step S17.

[0045] Next, in step S17, the three-dimensional position and orientation of the measurement object 7 last measured by the position and orientation measuring unit 11 is adopted as the three-dimensional position and orientation of the measurement object 7 for controlling the operation of the robot 2 with respect to the object to which the measurement object 7 is attached. Then, this process ends.

[0046] In the example of the flowchart in FIG. 3, the last measured position and orientation of the measurement object is adopted in step S17. However, as described above, when the amount of movement of the visual sensor 4 in the correction process is very small, from the viewpoint of the effect of the accuracy of the movement of the robot 2 itself on the correction error, the 3D position and orientation measured before the last one may be adopted.

[0047] Fig. 4 is a diagram showing a change in an input image when the imaging position is moved. Fig. 4 shows a change in an input image when the correction process by the imaging position correction unit 12 is repeatedly executed three times. Fig. 4 also shows an example in which a calibration jig is used as the measurement object 7.

[0048] Here, the calibration jig will be described in detail. As the calibration jig, a conventionally known calibration jig that can be used for calibrating the visual sensor 4 can be used. The calibration jig is a jig for acquiring information required for calibrating the visual sensor 4 by imaging a dot pattern arranged on a plane with the visual sensor 4, and satisfies three requirements for a dot pattern: (1) the lattice point interval of the dot pattern is known, (2) there is a certain number of lattice points or more, and (3) it is possible to uniquely identify which lattice point corresponds to which lattice point. The calibration jig is not limited to one in which a predetermined dot pattern or other features are arranged on a two-dimensional plane as shown in FIG. 4, but may be one in which features are arranged on a three-dimensional solid, and may be one that can obtain three-dimensional position information including position information in the height direction (Z direction) in addition to two-dimensional position information (X direction, Y direction). In addition, this calibration jig may be the same as or different from the one used when performing the calibration of the visual sensor 4. In order to calculate the three-dimensional position and orientation of a dot pattern captured by the visual sensor 4 with respect to the visual sensor 4, the internal parameters of the calibration data described above are used.

[0049] 4, when the correction process is performed by the imaging position correction unit 12, the relative position between the imaging position and the measurement object 7 changes, and therefore the three-dimensional position and orientation of the measurement object 7 in the input image gradually changes. It can also be seen that by performing the correction process multiple times, the amount of movement of the visual sensor 4 approaches 0, and the position of the measurement object 7 in the input image converges to the center of the input image.

[0050] 5 is a diagram showing the position of the measurement object 7 before the imaging position is moved and the position S of the measurement object 7 when the reference position is set. FIG. 6 is a diagram showing the position of the measurement object 7 after the imaging position is moved and the position S of the measurement object 7 when the reference position is set. As can be seen from FIGS. 5 and 6, by executing the correction process by the imaging position correction unit 12, the position of the measurement object 7 in the input image converges to the position S of the measurement object 7 when the reference position in the center of the input image is set. In other words, it can be seen that the relative position between the current imaging position and the measurement object 7 approaches the relative position between the imaging position and the measurement object 7 when the reference position of the robot 2 is set.

[0051] The position and orientation measurement system 1 according to the first embodiment provides the following advantages.

[0052] In the position and orientation measurement system 1 according to the present embodiment, an imaging position correction unit 12 is provided which uses the three-dimensional position and orientation of the measurement object 7 measured by the position and orientation measurement unit 11 to move the robot 2 to a position of the robot 2 where the visual sensor 4 approaches the position of the visual sensor 4 as seen from the measurement object 7 at the time of setting the reference position by the position and orientation moving unit 14, and then executes at least one correction process in which the image processing unit 21 processes the image and the position and orientation measurement unit 11 measures the image in sequence. In addition, in the position and orientation measurement system 1 according to the present embodiment, an imaging position correction end unit 13 is provided which executes the correction process again when it is determined that the correction process by the imaging position correction unit 12 has not been executed a predetermined number of times and the correction process is not to be ended, and which adopts any one of the three-dimensional position and orientation of the measurement object 7 calculated last or midway by the position and orientation measurement unit 11 as the three-dimensional position and orientation of the measurement object 7 to be used for controlling the robot 2 when it is determined that the correction process by the imaging position correction unit 12 has not been executed a predetermined number of times and is not to be ended.

[0053] As a result, the correction process by the imaging position correction unit 12 brings the current relative position between the imaging position and the measurement object 7 closer to the relative position between the imaging position and the measurement object 7 when the reference position of the robot 2 was set. Therefore, according to this embodiment, the variation in the measurement of the three-dimensional position and orientation of the measurement object 7 by the position and orientation measurement unit 11 can be suppressed, so that the three-dimensional position and orientation of the measurement object 7 can be accurately measured with simpler teaching and in a shorter measurement time than in the past. Furthermore, according to this embodiment, the measured three-dimensional position and orientation of the measurement object 7 can be used for work that requires precise operation of the robot 2.

[0054] [Modification of the first embodiment] The position and orientation measurement system according to the modification of the first embodiment has the same configuration as that of the first embodiment, except that the configuration of the imaging position correction ending unit 13 is different from that of the position and orientation measurement system 1 according to the first embodiment. Specifically, the imaging position correction ending unit according to the modification of the first embodiment determines whether or not to end the correction process depending on the amount of movement of the visual sensor 4 in the correction process by the imaging position correction unit 12. That is, the imaging position correction ending unit according to the modification of the first embodiment determines not to end the correction process when the amount of movement of the visual sensor 4 by the imaging position correction unit 12 is greater than a predetermined threshold, and determines to end the correction process when the amount of movement of the visual sensor 4 is smaller than the predetermined threshold.

[0055] Fig. 7 is a flowchart showing the procedure of processing in a position and orientation measuring system according to a modified example of the first embodiment. Steps S21 to S25 in Fig. 7 are similar to steps S11 to S15 in the first embodiment.

[0056] In step S26, it is determined whether the amount of movement of the visual sensor 4 is smaller than a predetermined threshold. Specifically, it is determined whether the amount of movement of the visual sensor 4 in step S23 is smaller than a predetermined threshold. The amount of movement of the visual sensor 4 is calculated from the amount of movement of the robot 2. If this determination is NO, the process returns to step S23, and the image capture position correction unit 12 again executes the correction process, which is composed of the movement of the visual sensor 4, the captured image processing, and the 3D position and orientation measurement. If this determination is YES, the process proceeds to step S27.

[0057] Next, in step S27, similarly to step S17 in the first embodiment, the three-dimensional position and orientation of the measurement object 7 last measured by the position and orientation measurement unit 11 is adopted as the three-dimensional position and orientation of the measurement object 7 used to control the robot 2. Then, this process ends.

[0058] In the example of the flowchart in FIG. 7, the last measured position and orientation of the measurement object is adopted in step S27. However, as in the first embodiment, when the amount of movement of the visual sensor 4 in the correction process is very small, from the viewpoint of the effect of the accuracy of the movement of the robot 2 itself on the correction error, for example, the 3D position and orientation measured before the last may be adopted.

[0059] Also, if the amount of movement of the visual sensor 4 does not become smaller than a predetermined threshold, an alarm may be issued to notify the user. In this case, it is preferable to set an upper limit on the number of times the correction process is performed in advance.

[0060] The position and orientation measurement system according to the modification of the first embodiment provides the same effects as the position and orientation measurement system 1 according to the first embodiment. In addition, according to this modification, when the amount of movement of the visual sensor 4 from the first correction process is small, the correction process can be performed in a shorter time than in the first embodiment.

[0061] [Second embodiment] Fig. 8 is a block diagram showing the configuration of a position and orientation measurement system 1A according to the second embodiment. As shown in Fig. 8, the position and orientation measurement system 1A according to the second embodiment has the same configuration as that of the first embodiment, except that the configuration of a visual sensor control device 20A is different from that of the position and orientation measurement system 1 according to the first embodiment. Specifically, in the position and orientation measurement system 1A according to the second embodiment, the visual sensor control device 20A further includes a measurement target luminance determination unit 24 and a measurement target luminance adjustment unit 25.

[0062] The measurement object luminance determination unit 24 measures the luminance of the measurement object 7 in the captured image processed by the image processing unit 21. The measurement object luminance determination unit 24 also determines whether or not the difference between the measured luminance and the luminance of the measurement object 7 in the captured image when the robot 2 is set to the reference position is greater than a predetermined first threshold value.

[0063] When the measurement object luminance determination unit 24 determines that the difference in luminance is greater than a predetermined first threshold, the measurement object luminance adjustment unit 25 adjusts the luminance of the measurement object 7 so that it approaches the luminance when the reference position of the robot 2 is set. Specifically, the measurement object luminance adjustment unit 25 changes the exposure conditions of the visual sensor 4 or combines multiple captured images with different exposure conditions of the visual sensor 4 so that the luminance of the measurement object 7 approaches the luminance when the reference position of the robot 2 is set.

[0064] The processing by the measurement target luminance determination unit 24 and the measurement target luminance adjustment unit 25 is executed together with the correction processing by the imaging position correction unit 12. That is, after the visual sensor 4 moves, the processing by the measurement target luminance determination unit 24 and the measurement target luminance adjustment unit 25 is executed in parallel with the captured image processing and the 3D position and orientation measurement.

[0065] 9 is a diagram showing a change in an input image when the imaging position is moved and the brightness is adjusted. In the example shown in FIG. 9, the brightness of the measurement object 7 in the input image after the first imaging position movement is lower than the brightness of the measurement object 7 in the input image when the reference position is set, and the difference between these brightnesses exceeds a first threshold. Therefore, the measurement object brightness adjustment unit 25 changes the exposure conditions and adjusts the brightness when the imaging position is moved for the second time. As a result, it can be seen that the brightness of the measurement object 7 in the input image after the second imaging position movement is equivalent to the brightness of the measurement object 7 in the input image when the reference position is set.

[0066] According to the position and orientation measurement system 1A of the second embodiment, when capturing an image at a moved capturing position, the exposure conditions of the visual sensor 4 are automatically adjusted so that the luminance, i.e., brightness, of the measurement object 7 in the captured image approaches the brightness when the reference position is set. Alternatively, in this case, an image is synthesized from a plurality of captured images with different luminance. The image synthesis is performed by a conventionally known HDR synthesis or the like. This makes it possible to bring the brightness of the measurement object 7 in the captured image closer to that when the reference position is set and when capturing an image after the visual sensor 4 is moved, thereby suppressing an increase in measurement error of the three-dimensional position and orientation of the measurement object 7 caused by the brightness of the measurement object 7 differing from that when the reference position is set.

[0067] [Third embodiment] Fig. 10 is a block diagram showing the configuration of a position and orientation measurement system 1B according to the third embodiment. As shown in Fig. 10, the position and orientation measurement system 1B according to the third embodiment has the same configuration as the first embodiment, except that it is different from the position and orientation measurement system 1 according to the first embodiment in that it has a different configuration of a visual sensor control device 20B and further includes an interface device 30.

[0068] Specifically, in a position and orientation measurement system 1B according to the third embodiment, a visual sensor control device 20B further includes a measurement object luminance determination unit 24. This measurement object luminance determination unit 24 has a configuration basically similar to that of the third embodiment.

[0069] The measurement object luminance determination unit 24 measures the luminance of the measurement object 7 in the captured image processed by the image processing unit 21. The measurement object luminance determination unit 24 also determines whether or not the difference between the luminance of the measurement object 7 in the captured image when the reference position of the robot 2 is set is greater than a predetermined second threshold value. Here, the predetermined second threshold value may be set to the same value as the above-mentioned first threshold value, or may be set to a value different from the first threshold value.

[0070] The interface device 30 includes a measurement target information presenting unit 31. When the measurement target luminance determining unit 24 determines that the difference in luminance is greater than a predetermined second threshold, the measurement target information presenting unit 31 presents to the user that the luminance of the measurement target 7 is significantly different from when the reference position of the robot 2 was set. For example, the measurement target information presenting unit 31 can display in a pop-up format on the display screen of the interface device 30, "The brightness of the measurement target (marker) is significantly different from when the reference position was set. We recommend that you change the lighting environment."

[0071] The measurement target information presenting unit 31 is not limited to being provided in the interface device 30. For example, instead of the interface device 30, the measurement target information presenting unit may be provided in the display unit of the visual sensor control device 20B.

[0072] Furthermore, the processing by the measurement target luminance determination unit 24 and the measurement target information presentation unit 31 is executed together with the correction processing by the imaging position correction unit 12. That is, after the visual sensor 4 moves, the processing by the measurement target luminance determination unit 24 and the measurement target information presentation unit 31 is executed in parallel with the captured image processing and the 3D position and orientation measurement.

[0073] Fig. 11 is a diagram showing the display screen of the interface device 30 when it is determined that the difference in luminance of the measurement object before and after the movement of the imaging position is greater than the second threshold. In the example shown in Fig. 11, the luminance of the measurement object 7 in the input image after the imaging position is moved is lower than the luminance of the measurement object 7 in the input image when the reference position is set, and the difference in luminance between them exceeds the second threshold. Therefore, it can be seen that the measurement object information presenting unit 31 displays in the display screen of the interface device 30 in the form of a pop-up message saying, "The brightness of the marker is significantly different from when the reference position is set. We recommend that you change the lighting environment."

[0074] According to the position and orientation measurement system 1B of the third embodiment, when the brightness of the measurement object 7 is significantly different from when the reference position was set, that information can be presented to the user. This makes it possible to prompt the user to change the lighting environment and manually change the exposure conditions of the visual sensor 4, thereby suppressing an increase in measurement errors of the three-dimensional position and orientation of the measurement object 7 caused by changes in the lighting conditions due to external factors.

[0075] [Fourth embodiment] Fig. 12 is a block diagram showing the configuration of a position and orientation measurement system 1C according to the fourth embodiment. As shown in Fig. 12, the position and orientation measurement system 1C according to the fourth embodiment has the same configuration as the first embodiment, except that it is different from the position and orientation measurement system 1 according to the first embodiment in that it has a different configuration of a visual sensor control device 20C and further includes an interface device 30.

[0076] Specifically, in the position and orientation measurement system 1C according to the fourth embodiment, the visual sensor control device 20C further includes a measurement target undetectable range determination unit 26. The measurement target undetectable range determination unit 26 measures the undetectable range of the measurement target 7 in the captured image processed by the image processing unit 21, and determines whether or not the measured value is greater than a predetermined undetectable threshold.

[0077] Here, the undetectable range of the measurement object 7 in the captured image means a range in which the measurement object 7 cannot be detected due to, for example, dust, some kind of obstruction, or reflection or halation caused by lighting provided on the visual sensor 4. The undetectable range of the measurement object 7 can be measured, for example, from the ratio of the area of ​​the undetectable range of the measurement object 7 to the area of ​​the entire captured image.

[0078] The interface device 30 includes a measurement target information presenting unit 31. When the measurement target undetectable range determining unit 26 determines that the measurement value of the undetectable range is greater than a predetermined undetectable threshold, the measurement target information presenting unit 31 presents to the user that the undetectable range is large. For example, the measurement target information presenting unit 31 can display in pop-up format on the display screen of the interface device 30, "The undetectable range of the measurement target (marker) is large. Please change the lighting environment or check whether the measurement target is hidden by an obstruction."

[0079] The measurement target information presentation unit 31 is not limited to being provided in the interface device 30. For example, instead of the interface device 30, the measurement target information presentation unit may be provided in the display unit of the visual sensor control device 20C.

[0080] Furthermore, the processing by the measurement target undetectable range determination unit 26 and the measurement target information presentation unit 31 is executed together with the correction processing by the imaging position correction unit 12. That is, after the visual sensor 4 is moved, the processing by the measurement target undetectable range determination unit 26 and the measurement target information presentation unit 31 is executed in parallel with the captured image processing and the 3D position and orientation measurement.

[0081] Fig. 13 is a diagram showing the display screen of the interface device 30 when it is determined that the undetectable range of the measurement object 7 in the input image after the imaging position has been moved is greater than the undetectable threshold. In the example shown in Fig. 13, obstructions 8 and 9 overlap the measurement object 7 in the input image after the imaging position has been moved, and the undetectable range exceeds the undetectable threshold. Therefore, it can be seen that the measurement object information presenting unit 31 displays in pop-up form on the display screen of the interface device 30, "The undetectable range of the marker is large. Please change the lighting environment or check whether the marker is hidden by an obstruction."

[0082] According to the position and orientation measurement system 1C of the fourth embodiment, when a part of the measurement object 7 is hidden by an external factor such as dust or some kind of obstruction, or when the measurement object 7 cannot be detected in the captured image due to reflection or halation caused by lighting, the undetectable range is measured, and when the undetectable range is larger than a predetermined undetectable threshold, information for alerting the user can be presented. This makes it possible to prompt the user to change the lighting environment so as to reduce the undetectable range of the measurement object 7, and to prompt the user to remove the external factor. Therefore, according to this embodiment, it is possible to further suppress an increase in measurement error of the 3D position and orientation of the measurement object 7 caused by an increase in the undetectable range of the measurement object 7.

[0083] The present disclosure is not limited to the above-described aspects, and modifications and improvements within the scope that can achieve the object of the present disclosure are included in the present disclosure.

[0084] For example, unlike the above embodiment, the position and orientation measurement system of the present disclosure can be applied to a system in which the robot 2 holds the measurement object 7 and the visual sensor 4 is attached to a machine tool. In this case, the imaging position correction unit 12 executes the correction process by moving the measurement object 7 using the position and orientation moving unit 14. Also, the imaging position correction ending unit 13 executes the end determination of the correction process based on the amount of movement of the measurement object 7.

[0085] It is also possible to attach the visual sensor 4 to the hands of different robots, for example, and use the visual sensor 4 to execute the position and orientation measurement system of the present disclosure with two robots.

[0086] Furthermore, the imaging position correction unit 12 in the above embodiment may be configured to execute the image processing by the image processing unit 21 and the correction process based on the measurement by the position and orientation measurement unit 11 at least once after moving the visual sensor 4 or the measurement object 7 so as to approach a first position and orientation indicating a reference orientation of the measurement object 7 (i.e., a position and orientation at the time of setting a reference position for correction). In this case, the imaging position correction termination unit 13 in the above embodiment may be configured to determine whether to terminate the correction process based on the second position and orientation of the measurement object 7 measured by the position and orientation measurement unit 11 (i.e., any of the positions and orientations measured by the position and orientation measurement unit 11 by the correction process) during the correction process by the imaging position correction unit 12. [Explanation of symbols]

[0087] 1,1A,1B,1C Position and Orientation Measurement System 2. Robot 3. Robot Arm 4. Visual Sensor 5 Flange 6. Hand Tools 7 Measurement Object 10 Robot control device 11 Position and Orientation Measurement Unit 12 Imaging position correction unit 13. Imaging position correction end section 14 Position / posture moving unit 20, 20A, 20B, 20C Visual sensor control device 21 Image processing section 22 Calibration data storage unit 23 Image acquisition unit 24 Measurement object luminance judgment unit 25 Measurement target brightness adjustment section 26 Measurement object undetectable range determination unit 30 Interface device 31 Measurement target information presentation unit 100 Robot System S Position when setting reference position T unit

Claims

1. A position and orientation measurement system that measures a three-dimensional position and orientation of a measurement object used for controlling a robot, comprising: A calibrated visual sensor; a position and orientation moving unit capable of moving a three-dimensional position and orientation of the measurement object or the visual sensor; a calibration data storage unit that stores in advance calibration data obtained when the visual sensor is calibrated; an image processing unit that processes an image obtained by capturing an image of the measurement object by the visual sensor based on the calibration data; a position and orientation measurement unit that measures a three-dimensional position and orientation of the measurement object using a result of the image processing by the image processing unit; an imaging position correction unit that performs a correction process at least once, in which the visual sensor or the measurement object is moved by the position and orientation moving unit using the three-dimensional position and orientation of the measurement object measured by the position and orientation measurement unit, and then the image processing unit and the measurement unit are sequentially performed; an imaging position correction ending unit that determines whether or not the correction process by the imaging position correction unit is to be ended, and if it is determined that the correction process is not to be ended, executes the correction process again, and if it is determined that the correction process is to be ended, adopts any one of the three-dimensional position and orientation of the measurement object measured last or during by the position and orientation measurement unit as the three-dimensional position and orientation of the measurement object to be used for control of the robot, The position and orientation measurement system, wherein the measurement object is a calibration jig or a marker.

2. 2. The position and orientation measurement system according to claim 1, wherein the imaging position correction unit performs the correction process after moving the robot to a position where the visual sensor approaches a position of the visual sensor as seen from the measurement object at a time when a reference position for correction is set.

3. The position and orientation measurement system according to claim 1 , wherein the imaging position correction end unit determines to end the correction process when the correction process by the imaging position correction unit has been executed a predetermined number of times.

4. 3 . The position and orientation measurement system according to claim 1 , wherein the imaging position correction end unit determines to end the correction process when an amount of movement of the visual sensor or the measurement object by the imaging position correction unit is smaller than a predetermined threshold value.

5. a measurement object brightness determination unit that measures the brightness of the measurement object in the captured image processed by the image processing unit, and determines whether or not a difference between the brightness of the measurement object in the captured image at the time of setting a reference position of the robot and the brightness of the measurement object in the captured image is greater than a predetermined first threshold value; 5. The position and orientation measurement system according to claim 1, further comprising a measurement object luminance adjustment unit that adjusts the luminance of the measurement object so as to approach a luminance at a time when a reference position of the robot is set, when the measurement object luminance determination unit determines that the luminance difference is greater than the predetermined first threshold value.

6. a measurement object brightness determination unit that measures the brightness of the measurement object in the captured image processed by the image processing unit, and determines whether or not a difference between the brightness of the measurement object in the captured image at the time of setting the reference position of the robot and the brightness of the measurement object in the captured image is greater than a predetermined second threshold value; 6. The position and orientation measuring system according to claim 1, further comprising: a measurement object information presenting unit that, when the measurement object luminance determination unit determines that the difference in luminance is greater than the predetermined second threshold, presents a user with a message that the luminance of the measurement object is significantly different from that at the time when the reference position of the robot was set.

7. a measurement object undetectable range determination unit that measures an undetectable range of the measurement object in the captured image processed by the image processing unit and determines whether the undetectable range is greater than a predetermined undetectable threshold; 7. The position and orientation measuring system according to claim 1, further comprising: a measurement target information presenting unit that, when the measurement target undetectable range determination unit determines that the undetectable range is larger than the predetermined undetectable threshold, presents to a user that the undetectable range of the measurement target is large.

8. A calibrated visual sensor; an image processing unit that processes an image of a measurement object obtained by imaging the measurement object with the visual sensor based on calibration data obtained when the visual sensor is calibrated; a position and orientation measurement unit that measures a position and orientation of the measurement object using a result of the image processing by the image processing unit; an imaging position correction unit that performs image processing by the image processing unit and correction processing based on the measurement by the position and orientation measurement unit at least once after moving the visual sensor or the measurement object so as to approach a first position and orientation that indicates a reference orientation of the measurement object; an imaging position correction ending unit that determines whether to end the correction process based on a second position and orientation of the measurement object measured by the position and orientation measurement unit during the correction process by the imaging position correction unit, The position and orientation measurement system, wherein the measurement object is a calibration jig or a marker.

9. A position and orientation measurement system for measuring a three-dimensional position and orientation of a measurement object used to control a robot, comprising: A calibrated visual sensor; a position and orientation moving unit capable of moving a three-dimensional position and orientation of the measurement object or the visual sensor; a calibration data storage unit that stores in advance calibration data obtained when the visual sensor is calibrated; an image processing unit that processes an image obtained by capturing an image of the measurement object by the visual sensor; a position and orientation measurement unit that measures a three-dimensional position and orientation of the measurement object using a result of the image processing by the image processing unit; an imaging position correction unit that performs a correction process at least once, in which the visual sensor or the measurement object is moved by the position and orientation moving unit using the three-dimensional position and orientation of the measurement object measured by the position and orientation measurement unit, and then the image processing unit and the measurement unit are sequentially performed; an imaging position correction ending unit that determines whether or not the correction process by the imaging position correction unit is to be ended, and if it is determined that the correction process is not to be ended, executes the correction process again, and if it is determined that the correction process is to be ended, adopts any one of the three-dimensional position and orientation of the measurement object measured last or during by the position and orientation measurement unit as the three-dimensional position and orientation of the measurement object to be used for control of the robot, a measurement object brightness determination unit that measures the brightness of the measurement object in the captured image processed by the image processing unit, and determines whether or not a difference between the brightness of the measurement object in the captured image at the time of setting a reference position of the robot and the brightness of the measurement object in the captured image is greater than a predetermined first threshold value; a measurement object luminance adjustment unit that adjusts the luminance of the measurement object so as to approach that at the time when a reference position of the robot is set, when the measurement object luminance determination unit determines that the luminance difference is greater than the predetermined first threshold value.

10. A position and orientation measurement system for measuring a three-dimensional position and orientation of a measurement object used to control a robot, comprising: A calibrated visual sensor; a position and orientation moving unit capable of moving a three-dimensional position and orientation of the measurement object or the visual sensor; a calibration data storage unit that stores in advance calibration data obtained when the visual sensor is calibrated; an image processing unit that processes an image obtained by capturing an image of the measurement object by the visual sensor; a position and orientation measurement unit that measures a three-dimensional position and orientation of the measurement object using a result of the image processing by the image processing unit; an imaging position correction unit that performs a correction process at least once, in which the visual sensor or the measurement object is moved by the position and orientation moving unit using the three-dimensional position and orientation of the measurement object measured by the position and orientation measurement unit, and then the image processing unit and the measurement unit are sequentially performed; an imaging position correction ending unit that determines whether or not the correction process by the imaging position correction unit is to be ended, and if it is determined that the correction process is not to be ended, executes the correction process again, and if it is determined that the correction process is to be ended, adopts any one of the three-dimensional position and orientation of the measurement object measured last or during by the position and orientation measurement unit as the three-dimensional position and orientation of the measurement object to be used for control of the robot, a measurement object brightness determination unit that measures the brightness of the measurement object in the captured image processed by the image processing unit, and determines whether or not a difference between the brightness of the measurement object in the captured image at the time of setting the reference position of the robot and the brightness of the measurement object in the captured image is greater than a predetermined second threshold value; a measurement object information presentation unit that, when the measurement object luminance determination unit determines that the difference in luminance is greater than the predetermined second threshold, presents a presentation to a user that the luminance of the measurement object is significantly different from that at the time when the robot was set at a reference position.

11. A position and orientation measurement system for measuring a three-dimensional position and orientation of a measurement object used to control a robot, comprising: A calibrated visual sensor; a position and orientation moving unit capable of moving a three-dimensional position and orientation of the measurement object or the visual sensor; a calibration data storage unit that stores in advance calibration data obtained when the visual sensor is calibrated; an image processing unit that processes an image obtained by capturing an image of the measurement object by the visual sensor; a position and orientation measurement unit that measures a three-dimensional position and orientation of the measurement object using a result of the image processing by the image processing unit; an imaging position correction unit that performs a correction process at least once, in which the visual sensor or the measurement object is moved by the position and orientation moving unit using the three-dimensional position and orientation of the measurement object measured by the position and orientation measurement unit, and then the image processing unit and the measurement unit are sequentially performed; an imaging position correction ending unit that determines whether or not the correction process by the imaging position correction unit is to be ended, and if it is determined that the correction process is not to be ended, executes the correction process again, and if it is determined that the correction process is to be ended, adopts any one of the three-dimensional position and orientation of the measurement object measured last or during by the position and orientation measurement unit as the three-dimensional position and orientation of the measurement object to be used for control of the robot, a measurement object undetectable range determination unit that measures an undetectable range of the measurement object in the captured image processed by the image processing unit and determines whether the undetectable range is greater than a predetermined undetectable threshold; a measurement object information presentation unit that, when the measurement object undetectable range determination unit determines that the undetectable range is larger than the predetermined undetectable threshold, presents a user with information that the undetectable range of the measurement object is large.

12. A position and orientation measurement system described in any one of claims 9 to 11, wherein the imaging position correction unit performs the correction process after moving the robot to a position where the visual sensor approaches the position of the visual sensor as seen from the measurement object at the time of setting a reference position for correction.

13. A position and orientation measurement system as described in any one of claims 9 to 12, wherein the imaging position correction termination unit determines that the correction process is to be terminated when the correction process by the imaging position correction unit has been executed a predetermined number of times.

14. A position and orientation measurement system described in any one of claims 9 to 13, wherein the imaging position correction termination unit determines to terminate the correction process when the amount of movement of the visual sensor or the object to be measured by the imaging position correction unit is smaller than a predetermined threshold value.

Images